2024-03-25T14:03:34.606 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T14:03:34.607 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287 --verbose -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/orig.config.yaml 2024-03-25T14:03:34.653 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T14:03:34.739 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621287' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLLiXIgngNtdUUxjOEdjf74oeU6tS2Dqa8Z23Rg++8KRtAagxxs2CR8ZaYaXIpIn8ME7NRK96dK8IlUl2h3juVg= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvUXbyBTSDKsGDyI5JnMlAwPUD+cGxsn5VWUYR714IAgvBMNY3l3MWwkqHEz6JahOYUCIE5jkKSzVxpToE2Kvc= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T14:03:34.739 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T14:03:34.740 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T14:03:34.740 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T14:03:34.741 INFO:teuthology.task.internal:Checking packages... 2024-03-25T14:03:34.765 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T14:03:34.765 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T14:03:34.765 INFO:teuthology.packaging:ref: None 2024-03-25T14:03:34.765 INFO:teuthology.packaging:tag: None 2024-03-25T14:03:34.765 INFO:teuthology.packaging:branch: main 2024-03-25T14:03:34.765 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:03:34.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T14:03:35.119 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T14:03:35.121 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T14:03:35.128 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T14:03:35.128 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T14:03:35.147 INFO:teuthology.task.internal:Saving configuration 2024-03-25T14:03:35.164 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T14:03:35.169 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:03:35.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:58:41.364631', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLLiXIgngNtdUUxjOEdjf74oeU6tS2Dqa8Z23Rg++8KRtAagxxs2CR8ZaYaXIpIn8ME7NRK96dK8IlUl2h3juVg='} 2024-03-25T14:03:35.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:58:41.365932', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvUXbyBTSDKsGDyI5JnMlAwPUD+cGxsn5VWUYR714IAgvBMNY3l3MWwkqHEz6JahOYUCIE5jkKSzVxpToE2Kvc='} 2024-03-25T14:03:35.220 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T14:03:35.226 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T14:03:35.226 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T14:03:35.226 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T14:03:35.297 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdc7abd20d0>, signals=[15]) 2024-03-25T14:03:35.298 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T14:03:35.304 INFO:teuthology.task.internal:Opening connections... 2024-03-25T14:03:35.304 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:03:35.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:03:35.413 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:03:35.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:03:35.501 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T14:03:35.508 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-03-25T14:03:35.525 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-03-25T14:03:35.526 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-25T14:03:35.582 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T14:03:35.583 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T14:03:35.584 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-25T14:03:35.603 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-03-25T14:03:35.620 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-03-25T14:03:35.620 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-03-25T14:03:35.676 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T14:03:35.677 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T14:03:35.678 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-03-25T14:03:35.703 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T14:03:35.713 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T14:03:35.718 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T14:03:35.718 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-03-25T14:03:35.721 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-03-25T14:03:35.736 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T14:03:35.742 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T14:03:35.742 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-03-25T14:03:35.779 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-03-25T14:03:35.874 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T14:03:35.938 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T14:03:35.956 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T14:03:35.957 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T14:03:35.957 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T14:03:35.957 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T14:03:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:03:35.973 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-03-25T14:03:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:03:35.974 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-25T14:03:36.029 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-361.el9.x86_64 2024-03-25T14:03:36.030 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-361.el9.x86_64 2024-03-25T14:03:36.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-03-25T14:03:36.032 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-361.el9.x86_64 2024-03-25T14:03:36.032 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-361.el9.x86_64 2024-03-25T14:03:36.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-03-25T14:03:38.183 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T14:03:38.296 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T14:03:39.821 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-25T14:03:39.842 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-25T14:03:40.649 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-25T14:03:41.014 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.0 MB 00:00 2024-03-25T14:03:42.467 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-25T14:03:46.309 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-25T14:03:47.321 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 19 MB 00:05 2024-03-25T14:03:47.902 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-25T14:03:51.732 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-25T14:03:53.156 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-03-25T14:03:56.324 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-03-25T14:04:01.449 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T14:04:03.681 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T14:04:03.681 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T14:04:03.772 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T14:04:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T14:04:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T14:04:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:04:03.783 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:04:03.783 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:04:03.783 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-03-25T14:04:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:04:03.784 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 76 M 2024-03-25T14:04:03.784 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 123 M 2024-03-25T14:04:03.784 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:04:04.525 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 67 kB/s | 44 kB 00:00 2024-03-25T14:04:24.857 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 727 kB/s | 15 MB 00:20 2024-03-25T14:04:43.571 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 720 kB/s | 27 MB 00:39 2024-03-25T14:04:53.612 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 688 kB/s | 33 MB 00:49 2024-03-25T14:04:53.612 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:04:53.613 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.5 MB/s | 76 MB 00:49 2024-03-25T14:04:54.301 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:04:54.758 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:04:54.758 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:04:55.772 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:04:55.772 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:04:58.374 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:04:59.015 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:04:59.104 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:05:01.993 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:05:02.130 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:05:07.233 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:05:07.272 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:05:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:05:43.367 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:05:43.367 INFO:teuthology.orchestra.run.smithi026.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T14:05:43.367 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:05:43.412 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:05:43.868 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:05:43.868 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:05:43.868 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:05:43.868 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:05:44.284 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:05:44.285 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:05:44.617 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:05:44.724 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:05:44.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:05:44.725 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T14:05:44.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T14:05:44.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T14:05:44.725 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T14:05:44.726 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T14:05:44.726 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:05:44.810 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:05:44.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:05:44.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 68.el9 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-03-25T14:05:44.848 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8264254 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T14:05:44.849 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T14:05:44.850 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-03-25T14:05:44.850 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:05:44.850 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T14:05:44.851 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T14:05:44.851 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-430.el9.x86_64 2024-03-25T14:05:44.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T14:05:45.467 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-03-25T14:05:46.841 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T14:05:46.879 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-03-25T14:05:46.881 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-25T14:05:46.897 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.c0RxHKbFe8 2024-03-25T14:05:46.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c0RxHKbFe8 2024-03-25T14:05:46.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.c0RxHKbFe8 2024-03-25T14:05:47.118 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.c0RxHKbFe8 is 6KB 2024-03-25T14:05:47.167 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.c0RxHKbFe8 2024-03-25T14:05:47.182 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T14:05:47.252 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T14:05:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T14:05:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T14:05:47.253 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T14:05:47.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T14:05:47.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-03-25T14:06:17.400 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T14:06:17.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-25T14:06:17.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:07:17.404 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T14:07:26.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-25T14:07:26.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:07:26.778 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-25T14:07:27.160 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-25T14:07:27.160 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T14:07:27.161 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-25T14:07:27.176 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-430.el9.x86_64 2024-03-25T14:07:27.176 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T14:07:27.176 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T14:07:27.176 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T14:07:28.177 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T14:07:28.177 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T14:07:28.251 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-03-25T14:07:28.292 DEBUG:teuthology.parallel:result is None 2024-03-25T14:08:13.022 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 80 kB/s | 21 MB 04:25 2024-03-25T14:08:17.878 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T14:08:18.065 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:01 ago on Mon 25 Mar 2024 02:08:17 PM UTC. 2024-03-25T14:08:20.018 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T14:08:20.018 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T14:08:20.103 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:08:20.108 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-03-25T14:08:20.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:08:20.110 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 76 M 2024-03-25T14:08:20.110 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 123 M 2024-03-25T14:08:20.110 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:08:20.571 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 129 kB/s | 44 kB 00:00 2024-03-25T14:08:24.929 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 3.2 MB/s | 15 MB 00:04 2024-03-25T14:08:45.534 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.1 MB/s | 27 MB 00:24 2024-03-25T14:08:47.750 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.2 MB/s | 33 MB 00:27 2024-03-25T14:08:47.750 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:47.750 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.7 MB/s | 76 MB 00:27 2024-03-25T14:08:48.414 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:08:48.773 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:08:48.773 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:08:49.741 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:08:49.741 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:08:52.190 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:08:52.901 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:08:52.975 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:08:55.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:08:55.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:09:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:09:00.695 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:09:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:09:34.590 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:09:34.590 INFO:teuthology.orchestra.run.smithi121.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T14:09:34.590 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:09:34.632 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:09:34.966 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:09:34.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:09:34.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:09:34.966 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:09:35.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:09:35.369 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:09:35.699 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:09:35.806 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:09:35.807 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:09:35.807 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T14:09:35.807 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T14:09:35.807 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T14:09:35.807 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T14:09:35.807 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T14:09:35.807 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:09:35.887 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:09:35.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:09:35.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-03-25T14:09:35.927 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-03-25T14:09:35.927 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T14:09:35.928 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T14:09:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T14:09:35.930 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T14:09:35.930 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-03-25T14:09:35.930 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:09:35.930 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T14:09:35.931 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T14:09:35.931 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-430.el9.x86_64 2024-03-25T14:09:35.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T14:09:36.538 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-03-25T14:09:37.921 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T14:09:37.957 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-03-25T14:09:37.960 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-03-25T14:09:37.975 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.7DuLodKy7T 2024-03-25T14:09:37.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7DuLodKy7T 2024-03-25T14:09:38.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.7DuLodKy7T 2024-03-25T14:09:38.176 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.7DuLodKy7T is 6KB 2024-03-25T14:09:38.224 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.7DuLodKy7T 2024-03-25T14:09:38.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T14:09:38.302 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T14:09:38.303 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T14:09:38.303 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T14:09:38.303 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T14:09:38.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T14:09:38.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-03-25T14:10:08.461 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T14:10:08.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-25T14:10:08.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:10:26.975 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-03-25T14:10:35.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-25T14:10:35.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:11:35.979 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T14:11:47.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-25T14:11:47.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:11:48.362 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-25T14:11:48.749 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-25T14:11:48.750 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T14:11:48.750 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-03-25T14:11:48.766 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-430.el9.x86_64 2024-03-25T14:11:48.766 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T14:11:48.766 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T14:11:48.766 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T14:11:49.767 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T14:11:49.768 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T14:11:49.860 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-03-25T14:11:49.897 DEBUG:teuthology.parallel:result is None 2024-03-25T14:11:49.897 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T14:11:49.905 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T14:11:49.905 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T14:11:49.908 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T14:11:49.927 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T14:11:49.934 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T14:11:49.944 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T14:11:49.945 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T14:11:49.966 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T14:11:49.994 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T14:11:50.005 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T14:11:50.006 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T14:11:50.029 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T14:11:50.064 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:11:50.068 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:11:50.078 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:11:50.082 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:11:50.084 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T14:11:50.094 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T14:11:50.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T14:11:50.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T14:11:50.153 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T14:11:50.164 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T14:11:50.164 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T14:11:50.191 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T14:11:50.210 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:11:50.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:11:50.335 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:11:50.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:11:50.462 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:11:50.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T14:11:50.530 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:11:50.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:11:50.647 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:11:50.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:11:50.736 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:11:50.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T14:11:50.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-03-25T14:11:50.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-03-25T14:11:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:11:50.888 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:11:51.271 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T14:11:51.280 INFO:teuthology.task.internal:Starting timer... 2024-03-25T14:11:51.280 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T14:11:51.289 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T14:11:51.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-03-25T14:11:51.373 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-03-25T14:11:51.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-03-25T14:11:51.425 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-03-25T14:11:51.427 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T14:11:51.428 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-03-25T14:11:51.459 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-03-25T14:11:51.459 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-03-25T14:11:51.497 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-03-25T14:11:51.497 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-03-25T14:11:51.498 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:11:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:11:51.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:11:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:11:51.566 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T14:11:51.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-03-25T14:11:51.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-03-25T14:11:51.639 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T14:11:51.648 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T14:11:51.665 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T14:11:51.667 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T14:16:12.359 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-03-25T14:16:12.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-25T14:16:12.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:16:12.457 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-25T14:16:12.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-25T14:16:12.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-25T14:16:12.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:16:12.631 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-25T14:16:12.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-25T14:16:12.714 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T14:16:12.726 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T14:16:12.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T14:16:12.727 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:16:12.730 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T14:16:12.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:16:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T14:16:12.776 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T14:16:12.786 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T14:16:12.803 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T14:16:12.806 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-03-25T14:16:12.820 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-03-25T14:16:12.832 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-03-25T14:16:12.836 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T14:16:12.845 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-03-25T14:16:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T14:16:12.862 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T14:16:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T14:16:12.903 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:16:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:16:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-25T14:16:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.929 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:16:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:16:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-25T14:16:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:16:12.933 INFO:teuthology.run_tasks:Running task install... 2024-03-25T14:16:12.942 DEBUG:teuthology.task.install:project ceph 2024-03-25T14:16:12.942 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T14:16:12.942 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T14:16:12.943 INFO:teuthology.task.install:Using flavor: default 2024-03-25T14:16:12.959 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T14:16:12.959 INFO:teuthology.task.install:extra packages: [] 2024-03-25T14:16:12.959 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T14:16:12.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:16:12.962 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T14:16:12.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:16:13.341 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T14:16:13.342 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T14:16:13.354 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T14:16:13.355 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T14:16:13.464 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T14:16:13.465 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:16:13.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T14:16:13.494 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T14:16:13.494 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T14:16:13.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T14:16:13.639 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T14:16:13.640 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:16:13.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T14:16:13.644 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T14:16:13.672 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T14:16:13.672 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T14:16:13.712 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-03-25T14:16:13.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-03-25T14:16:13.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T14:16:13.826 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T14:16:13.894 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-03-25T14:16:13.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-03-25T14:16:13.963 INFO:teuthology.orchestra.run.smithi121.stdout:74 files removed 2024-03-25T14:16:13.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-03-25T14:16:14.145 INFO:teuthology.orchestra.run.smithi026.stdout:74 files removed 2024-03-25T14:16:14.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-03-25T14:16:14.834 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 149 kB/s | 82 kB 00:00 2024-03-25T14:16:15.034 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 147 kB/s | 82 kB 00:00 2024-03-25T14:16:15.168 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 35 kB/s | 10 kB 00:00 2024-03-25T14:16:15.360 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 37 kB/s | 10 kB 00:00 2024-03-25T14:16:15.702 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-25T14:16:15.743 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-25T14:16:16.371 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T14:16:16.412 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-25T14:16:18.046 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T14:16:18.104 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T14:16:18.898 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T14:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-25T14:16:22.695 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T14:16:25.607 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-25T14:16:27.883 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T14:16:38.616 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 1.9 MB/s | 21 MB 00:10 2024-03-25T14:16:43.557 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T14:16:45.068 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 906 kB/s | 19 MB 00:21 2024-03-25T14:16:45.962 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-03-25T14:16:45.969 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T14:16:45.970 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T14:16:45.971 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:16:45.972 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:16:45.973 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:16:45.973 INFO:teuthology.orchestra.run.smithi121.stdout:Install 30 Packages 2024-03-25T14:16:45.973 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-03-25T14:16:45.973 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:16:45.973 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 58 M 2024-03-25T14:16:45.974 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:16:46.975 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 7.7 MB/s | 5.3 MB 00:00 2024-03-25T14:16:47.042 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 25 kB 00:00 2024-03-25T14:16:47.133 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-25T14:16:47.225 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-25T14:16:47.776 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.3 MB/s | 5.1 MB 00:00 2024-03-25T14:16:47.976 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 12 MB/s | 20 MB 00:01 2024-03-25T14:16:48.001 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 200 kB/s | 45 kB 00:00 2024-03-25T14:16:48.026 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-25T14:16:48.052 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-25T14:16:48.085 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.3 MB/s | 320 kB 00:00 2024-03-25T14:16:48.111 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-25T14:16:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-25T14:16:48.252 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-03-25T14:16:48.278 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-25T14:16:48.303 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-25T14:16:48.354 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T14:16:48.379 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T14:16:48.404 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-25T14:16:48.629 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 4.2 MB/s | 9.7 MB 00:02 2024-03-25T14:16:49.105 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 50 kB/s | 35 kB 00:00 2024-03-25T14:16:49.144 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T14:16:49.288 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 68 kB/s | 44 kB 00:00 2024-03-25T14:16:49.389 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 214 kB/s | 216 kB 00:01 2024-03-25T14:16:49.439 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 926 kB/s | 309 kB 00:00 2024-03-25T14:16:49.556 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-25T14:16:49.622 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 111 kB/s | 26 kB 00:00 2024-03-25T14:16:49.648 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-03-25T14:16:49.923 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-03-25T14:16:50.182 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-03-25T14:16:50.407 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-03-25T14:16:50.482 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-03-25T14:16:50.611 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-25T14:16:50.799 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.5 MB/s | 3.4 MB 00:00 2024-03-25T14:16:51.849 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 3.1 MB 00:01 2024-03-25T14:16:51.854 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:51.854 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.8 MB/s | 58 MB 00:05 2024-03-25T14:16:51.889 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:16:51.927 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:16:51.927 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:16:52.121 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T14:16:52.404 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:16:52.404 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:16:53.848 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:16:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T14:16:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T14:16:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T14:16:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:16:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:16:54.230 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:16:54.282 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:16:54.619 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T14:16:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:16:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T14:16:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:16:54.920 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:16:54.949 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T14:16:54.992 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T14:16:55.060 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:16:55.084 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:16:55.147 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:16:55.174 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T14:16:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T14:16:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T14:16:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T14:16:55.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T14:16:55.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T14:16:55.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T14:16:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T14:16:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:16:55.880 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:16:55.912 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T14:16:55.956 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T14:16:56.411 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T14:16:56.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:16:56.498 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:16:56.565 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T14:16:56.595 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T14:16:57.180 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-25T14:16:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:16:58.791 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:16:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:16:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:16:59.704 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:16:59.707 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:16:59.707 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T14:16:59.708 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse 2024-03-25T14:16:59.709 INFO:teuthology.orchestra.run.smithi026.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:16:59.710 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:16:59.711 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:16:59.711 INFO:teuthology.orchestra.run.smithi026.stdout:Install 30 Packages 2024-03-25T14:16:59.711 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-03-25T14:16:59.711 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:16:59.711 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 58 M 2024-03-25T14:16:59.711 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:16:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:16:59.712 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T14:16:59.712 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T14:16:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:16:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:17:02.039 INFO:teuthology.orchestra.run.smithi026.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.7 MB/s | 5.3 MB 00:01 2024-03-25T14:17:02.139 INFO:teuthology.orchestra.run.smithi026.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T14:17:02.381 INFO:teuthology.orchestra.run.smithi026.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.0 MB/s | 742 kB 00:00 2024-03-25T14:17:02.573 INFO:teuthology.orchestra.run.smithi026.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.5 MB/s | 498 kB 00:00 2024-03-25T14:17:03.815 INFO:teuthology.orchestra.run.smithi026.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 4.1 MB/s | 5.1 MB 00:01 2024-03-25T14:17:03.906 INFO:teuthology.orchestra.run.smithi026.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 494 kB/s | 45 kB 00:00 2024-03-25T14:17:04.007 INFO:teuthology.orchestra.run.smithi026.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.3 MB/s | 131 kB 00:00 2024-03-25T14:17:04.115 INFO:teuthology.orchestra.run.smithi026.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.5 MB/s | 162 kB 00:00 2024-03-25T14:17:04.241 INFO:teuthology.orchestra.run.smithi026.stdout:(9/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 9.7 MB 00:04 2024-03-25T14:17:04.266 INFO:teuthology.orchestra.run.smithi026.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 320 kB 00:00 2024-03-25T14:17:04.376 INFO:teuthology.orchestra.run.smithi026.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 900 kB/s | 99 kB 00:00 2024-03-25T14:17:04.402 INFO:teuthology.orchestra.run.smithi026.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 298 kB 00:00 2024-03-25T14:17:04.527 INFO:teuthology.orchestra.run.smithi026.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-25T14:17:04.552 INFO:teuthology.orchestra.run.smithi026.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T14:17:04.578 INFO:teuthology.orchestra.run.smithi026.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-25T14:17:04.628 INFO:teuthology.orchestra.run.smithi026.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T14:17:04.653 INFO:teuthology.orchestra.run.smithi026.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T14:17:04.679 INFO:teuthology.orchestra.run.smithi026.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-25T14:17:04.796 INFO:teuthology.orchestra.run.smithi026.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 299 kB/s | 35 kB 00:00 2024-03-25T14:17:04.863 INFO:teuthology.orchestra.run.smithi026.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T14:17:04.888 INFO:teuthology.orchestra.run.smithi026.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 485 kB/s | 44 kB 00:00 2024-03-25T14:17:05.138 INFO:teuthology.orchestra.run.smithi026.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T14:17:05.197 INFO:teuthology.orchestra.run.smithi026.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-03-25T14:17:05.272 INFO:teuthology.orchestra.run.smithi026.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-03-25T14:17:05.339 INFO:teuthology.orchestra.run.smithi026.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-03-25T14:17:05.364 INFO:teuthology.orchestra.run.smithi026.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 738 kB/s | 67 kB 00:00 2024-03-25T14:17:05.406 INFO:teuthology.orchestra.run.smithi026.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T14:17:05.431 INFO:teuthology.orchestra.run.smithi026.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T14:17:05.498 INFO:teuthology.orchestra.run.smithi026.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-25T14:17:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 3.4 MB 00:00 2024-03-25T14:17:05.990 INFO:teuthology.orchestra.run.smithi026.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.4 MB/s | 3.1 MB 00:00 2024-03-25T14:17:06.766 INFO:teuthology.orchestra.run.smithi026.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.0 MB/s | 20 MB 00:06 2024-03-25T14:17:06.769 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:06.770 INFO:teuthology.orchestra.run.smithi026.stdout:Total 8.2 MB/s | 58 MB 00:07 2024-03-25T14:17:06.802 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:17:06.844 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:17:06.844 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:17:07.338 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:17:07.339 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:17:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:17:09.043 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T14:17:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T14:17:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T14:17:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:17:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-03-25T14:17:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-03-25T14:17:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:09.332 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:17:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:17:09.769 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T14:17:09.814 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:17:09.923 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:17:10.012 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T14:17:10.040 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:17:10.074 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:17:10.086 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:17:10.098 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:17:10.099 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:10.099 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T14:17:10.099 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:17:10.099 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:17:10.099 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:10.102 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:17:10.115 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T14:17:10.159 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T14:17:10.194 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:17:10.194 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:17:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:17:10.251 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:17:10.313 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:17:10.341 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T14:17:10.452 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T14:17:10.488 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T14:17:10.525 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T14:17:10.557 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T14:17:10.586 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T14:17:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T14:17:10.943 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T14:17:10.973 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:17:11.064 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:17:11.095 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T14:17:11.141 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T14:17:11.319 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:17:11.319 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T14:17:11.319 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T14:17:11.319 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T14:17:11.319 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T14:17:11.319 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T14:17:11.320 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:17:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T14:17:11.322 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T14:17:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T14:17:11.654 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:17:11.731 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:17:11.801 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T14:17:11.845 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T14:17:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T14:17:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:12.275 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:17:12.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-03-25T14:17:13.184 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:30 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:17:13.338 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:17:13.339 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:13.340 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:17:13.341 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:17:13.341 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-03-25T14:17:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:13.341 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 48 M 2024-03-25T14:17:13.341 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 204 M 2024-03-25T14:17:13.341 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:17:13.524 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T14:17:13.575 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-25T14:17:13.616 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T14:17:13.800 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T14:17:14.050 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 245 kB/s | 163 kB 00:00 2024-03-25T14:17:14.103 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:17:14.141 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:17:14.725 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:17:14.762 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:17:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:17:14.853 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T14:17:14.853 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T14:17:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:14.888 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:17:25.114 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:17:25.114 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-03-25T14:17:25.114 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-03-25T14:17:25.114 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-03-25T14:17:25.115 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-03-25T14:17:25.115 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-03-25T14:17:25.115 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-03-25T14:17:25.115 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-03-25T14:17:25.115 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:26.048 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:17:26.095 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:17:26.096 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:26.096 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T14:17:26.096 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:17:26.096 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:17:26.096 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:26.096 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:17:26.168 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:17:26.168 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:17:27.269 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T14:17:27.270 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T14:17:27.271 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T14:17:27.272 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T14:17:27.272 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T14:17:27.272 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T14:17:27.272 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T14:17:27.272 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T14:17:27.273 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T14:17:27.273 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:17:27.273 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T14:17:27.273 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T14:17:28.073 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 47 MB 00:14 2024-03-25T14:17:28.076 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:28.076 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.3 MB/s | 48 MB 00:14 2024-03-25T14:17:28.091 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:17:28.112 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:17:28.112 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:17:28.267 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:17:28.267 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:28.267 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T14:17:28.268 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.269 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:28.270 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:17:28.489 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:17:28.490 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:17:28.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-03-25T14:17:29.081 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:17:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T14:17:29.200 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:32 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:17:29.203 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T14:17:29.243 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:17:29.282 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T14:17:29.309 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:17:29.364 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:17:29.365 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:17:29.365 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:17:29.365 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:17:29.366 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-25T14:17:29.367 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:29.367 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 48 M 2024-03-25T14:17:29.367 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 204 M 2024-03-25T14:17:29.367 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:17:29.722 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-25T14:17:29.831 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-25T14:17:29.881 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T14:17:30.048 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 307 kB/s | 163 kB 00:00 2024-03-25T14:17:30.181 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 623 kB/s | 187 kB 00:00 2024-03-25T14:17:33.823 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:17:33.908 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:17:35.648 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 7.7 MB/s | 47 MB 00:06 2024-03-25T14:17:35.651 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:35.651 INFO:teuthology.orchestra.run.smithi026.stdout:Total 7.7 MB/s | 48 MB 00:06 2024-03-25T14:17:35.665 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:17:35.688 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:17:35.688 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:17:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:17:36.089 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:17:36.341 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:17:36.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T14:17:36.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T14:17:36.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:17:36.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T14:17:36.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T14:17:36.690 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-16.el9.x86_64 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T14:17:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T14:17:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:36.717 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:17:36.766 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T14:17:36.807 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T14:17:36.847 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:17:36.889 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T14:17:36.909 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:17:37.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-03-25T14:17:37.581 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:54 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:17:37.737 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T14:17:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T14:17:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T14:17:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout:Install 36 Packages 2024-03-25T14:17:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:37.744 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-03-25T14:17:37.744 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 107 M 2024-03-25T14:17:37.744 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:17:38.361 INFO:teuthology.orchestra.run.smithi121.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-03-25T14:17:38.586 INFO:teuthology.orchestra.run.smithi121.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-03-25T14:17:38.862 INFO:teuthology.orchestra.run.smithi121.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 9.4 MB/s | 4.7 MB 00:00 2024-03-25T14:17:38.937 INFO:teuthology.orchestra.run.smithi121.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-25T14:17:39.046 INFO:teuthology.orchestra.run.smithi121.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-25T14:17:39.087 INFO:teuthology.orchestra.run.smithi121.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T14:17:39.129 INFO:teuthology.orchestra.run.smithi121.stdout:(7/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2024-03-25T14:17:39.155 INFO:teuthology.orchestra.run.smithi121.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 470 kB/s | 32 kB 00:00 2024-03-25T14:17:39.180 INFO:teuthology.orchestra.run.smithi121.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2024-03-25T14:17:39.205 INFO:teuthology.orchestra.run.smithi121.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-25T14:17:39.230 INFO:teuthology.orchestra.run.smithi121.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T14:17:39.281 INFO:teuthology.orchestra.run.smithi121.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T14:17:39.306 INFO:teuthology.orchestra.run.smithi121.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-25T14:17:39.331 INFO:teuthology.orchestra.run.smithi121.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T14:17:39.357 INFO:teuthology.orchestra.run.smithi121.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-25T14:17:39.566 INFO:teuthology.orchestra.run.smithi121.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 17 MB/s | 17 MB 00:00 2024-03-25T14:17:39.582 INFO:teuthology.orchestra.run.smithi121.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-25T14:17:39.691 INFO:teuthology.orchestra.run.smithi121.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 347 kB/s | 125 kB 00:00 2024-03-25T14:17:40.041 INFO:teuthology.orchestra.run.smithi121.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 491 kB/s | 172 kB 00:00 2024-03-25T14:17:40.317 INFO:teuthology.orchestra.run.smithi121.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-25T14:17:40.400 INFO:teuthology.orchestra.run.smithi121.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T14:17:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 236 kB/s | 18 kB 00:00 2024-03-25T14:17:40.551 INFO:teuthology.orchestra.run.smithi121.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-25T14:17:40.626 INFO:teuthology.orchestra.run.smithi121.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 260 kB/s | 19 kB 00:00 2024-03-25T14:17:40.701 INFO:teuthology.orchestra.run.smithi121.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-25T14:17:40.785 INFO:teuthology.orchestra.run.smithi121.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 555 kB/s | 46 kB 00:00 2024-03-25T14:17:40.869 INFO:teuthology.orchestra.run.smithi121.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 946 kB/s | 79 kB 00:00 2024-03-25T14:17:40.936 INFO:teuthology.orchestra.run.smithi121.stdout:(28/36): luarocks-3.9.2-1.el9.noarch.rpm 110 kB/s | 151 kB 00:01 2024-03-25T14:17:41.019 INFO:teuthology.orchestra.run.smithi121.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-03-25T14:17:41.044 INFO:teuthology.orchestra.run.smithi121.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-25T14:17:41.094 INFO:teuthology.orchestra.run.smithi121.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T14:17:41.120 INFO:teuthology.orchestra.run.smithi121.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-03-25T14:17:41.228 INFO:teuthology.orchestra.run.smithi121.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-25T14:17:41.312 INFO:teuthology.orchestra.run.smithi121.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-25T14:17:41.345 INFO:teuthology.orchestra.run.smithi121.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-03-25T14:17:41.618 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:17:41.702 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:17:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:17:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T14:17:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T14:17:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:17:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T14:17:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T14:17:44.465 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: jq-1.6-16.el9.x86_64 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:44.466 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:17:44.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-03-25T14:17:45.339 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:48 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:17:45.497 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T14:17:45.500 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T14:17:45.501 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T14:17:45.502 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout:Install 36 Packages 2024-03-25T14:17:45.503 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:45.504 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 30 M 2024-03-25T14:17:45.504 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 107 M 2024-03-25T14:17:45.504 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:17:46.201 INFO:teuthology.orchestra.run.smithi026.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-25T14:17:46.443 INFO:teuthology.orchestra.run.smithi026.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.2 MB/s | 2.1 MB 00:00 2024-03-25T14:17:46.501 INFO:teuthology.orchestra.run.smithi026.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 1.5 MB 00:00 2024-03-25T14:17:46.585 INFO:teuthology.orchestra.run.smithi026.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.9 MB/s | 247 kB 00:00 2024-03-25T14:17:46.702 INFO:teuthology.orchestra.run.smithi026.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T14:17:46.760 INFO:teuthology.orchestra.run.smithi026.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T14:17:46.794 INFO:teuthology.orchestra.run.smithi026.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-25T14:17:46.869 INFO:teuthology.orchestra.run.smithi026.stdout:(8/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.0 MB/s | 4.7 MB 00:00 2024-03-25T14:17:46.886 INFO:teuthology.orchestra.run.smithi026.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 512 kB/s | 47 kB 00:00 2024-03-25T14:17:46.903 INFO:teuthology.orchestra.run.smithi026.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T14:17:46.929 INFO:teuthology.orchestra.run.smithi026.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T14:17:46.979 INFO:teuthology.orchestra.run.smithi026.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T14:17:47.004 INFO:teuthology.orchestra.run.smithi026.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-25T14:17:47.028 INFO:teuthology.orchestra.run.smithi121.stdout:(36/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 5.8 kB/s | 43 kB 00:07 2024-03-25T14:17:47.029 INFO:teuthology.orchestra.run.smithi026.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T14:17:47.031 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:47.031 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.2 MB/s | 30 MB 00:09 2024-03-25T14:17:47.054 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:17:47.055 INFO:teuthology.orchestra.run.smithi026.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T14:17:47.095 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:17:47.095 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:17:47.147 INFO:teuthology.orchestra.run.smithi026.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-25T14:17:47.230 INFO:teuthology.orchestra.run.smithi026.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 623 kB/s | 125 kB 00:00 2024-03-25T14:17:47.397 INFO:teuthology.orchestra.run.smithi026.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 18 MB/s | 17 MB 00:00 2024-03-25T14:17:47.500 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:17:47.501 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:17:47.589 INFO:teuthology.orchestra.run.smithi026.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-03-25T14:17:47.606 INFO:teuthology.orchestra.run.smithi026.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 116 kB/s | 43 kB 00:00 2024-03-25T14:17:47.656 INFO:teuthology.orchestra.run.smithi026.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-25T14:17:47.681 INFO:teuthology.orchestra.run.smithi026.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T14:17:47.707 INFO:teuthology.orchestra.run.smithi026.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 607 kB/s | 172 kB 00:00 2024-03-25T14:17:47.724 INFO:teuthology.orchestra.run.smithi026.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2024-03-25T14:17:47.741 INFO:teuthology.orchestra.run.smithi026.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 393 kB/s | 23 kB 00:00 2024-03-25T14:17:47.757 INFO:teuthology.orchestra.run.smithi026.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-03-25T14:17:47.774 INFO:teuthology.orchestra.run.smithi026.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-03-25T14:17:47.791 INFO:teuthology.orchestra.run.smithi026.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-03-25T14:17:47.817 INFO:teuthology.orchestra.run.smithi026.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T14:17:47.842 INFO:teuthology.orchestra.run.smithi026.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-25T14:17:47.875 INFO:teuthology.orchestra.run.smithi026.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T14:17:47.894 INFO:teuthology.orchestra.run.smithi026.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-25T14:17:47.918 INFO:teuthology.orchestra.run.smithi026.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-25T14:17:47.960 INFO:teuthology.orchestra.run.smithi026.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-03-25T14:17:47.976 INFO:teuthology.orchestra.run.smithi026.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-25T14:17:48.002 INFO:teuthology.orchestra.run.smithi026.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-25T14:17:48.008 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:48.008 INFO:teuthology.orchestra.run.smithi026.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-25T14:17:48.033 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:17:48.079 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:17:48.079 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:17:48.513 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:17:48.514 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:17:48.849 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:17:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T14:17:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T14:17:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T14:17:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T14:17:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T14:17:49.123 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T14:17:49.217 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T14:17:49.293 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T14:17:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T14:17:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T14:17:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T14:17:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T14:17:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T14:17:49.620 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T14:17:49.804 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T14:17:49.838 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:17:49.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T14:17:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T14:17:49.896 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T14:17:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T14:17:49.940 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T14:17:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T14:17:49.975 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T14:17:50.024 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T14:17:50.047 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T14:17:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:17:50.091 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T14:17:50.091 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T14:17:50.124 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T14:17:50.131 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T14:17:50.166 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T14:17:50.201 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T14:17:50.219 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T14:17:50.290 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T14:17:50.295 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T14:17:50.332 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T14:17:50.343 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T14:17:50.414 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T14:17:50.417 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T14:17:50.470 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T14:17:50.507 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T14:17:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T14:17:50.565 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T14:17:50.629 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T14:17:50.676 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:17:50.820 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T14:17:50.832 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T14:17:50.862 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:17:50.870 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T14:17:50.887 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:17:50.887 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:50.887 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T14:17:50.887 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:17:50.887 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:17:50.888 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:50.909 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T14:17:50.952 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T14:17:50.981 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T14:17:51.060 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T14:17:51.092 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:17:51.125 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T14:17:51.165 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T14:17:51.201 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T14:17:51.235 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T14:17:51.327 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T14:17:51.383 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T14:17:51.457 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T14:17:51.597 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T14:17:51.730 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:17:51.898 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T14:17:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:17:51.954 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:17:51.954 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:51.954 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T14:17:51.954 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:17:51.955 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:17:51.955 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:52.338 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:17:52.364 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:17:52.364 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:52.365 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T14:17:52.365 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:17:52.365 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:17:52.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:52.805 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:17:52.828 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:17:52.828 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:52.828 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T14:17:52.828 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:17:52.829 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:17:52.829 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:53.039 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:17:53.062 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:17:53.062 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:53.062 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T14:17:53.062 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:17:53.062 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:17:53.063 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:53.128 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:17:53.471 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:17:53.497 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:17:53.498 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:53.498 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T14:17:53.498 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:17:53.498 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:17:53.498 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:53.955 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:17:53.978 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:17:53.978 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:53.979 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T14:17:53.979 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:17:53.979 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:17:53.979 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:17:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:17:54.229 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:17:54.229 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T14:17:54.229 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:17:54.229 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:17:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:54.298 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T14:17:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T14:17:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T14:17:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T14:17:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T14:17:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T14:17:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T14:17:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T14:17:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T14:17:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T14:17:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T14:17:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T14:17:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:55.755 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T14:17:56.089 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T14:17:56.090 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T14:17:56.091 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T14:17:56.092 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T14:17:56.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-03-25T14:17:56.672 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:13 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:17:56.769 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:17:56.821 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:17:56.822 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:17:56.822 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:17:56.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:57.069 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T14:17:57.070 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T14:17:57.071 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:57.072 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:17:57.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-03-25T14:17:57.437 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:14 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:17:57.586 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:17:57.586 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 747 k 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 754 k 2024-03-25T14:17:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:17:57.975 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.9 MB/s | 747 kB 00:00 2024-03-25T14:17:57.976 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:57.976 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.9 MB/s | 747 kB 00:00 2024-03-25T14:17:57.976 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:17:57.981 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:17:57.981 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:17:57.986 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:17:57.987 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:17:57.997 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:17:58.057 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:17:58.098 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:17:58.153 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:17:58.154 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:17:58.154 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:17:58.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-03-25T14:17:58.458 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:17:58.563 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:17:58.790 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:01 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:17:58.945 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:17:58.945 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:17:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 747 k 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 754 k 2024-03-25T14:17:58.946 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:17:58.985 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:17:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:17:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:59.251 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:17:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:17:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:17:59.251 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:17:59.388 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-25T14:17:59.388 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:59.389 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-25T14:17:59.389 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:17:59.394 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:17:59.394 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:17:59.400 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:17:59.401 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:17:59.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T14:17:59.473 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:17:59.938 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:18:00.021 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:18:00.049 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T14:18:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 436 k 2024-03-25T14:18:00.206 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:00.410 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:18:00.431 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 643 kB/s | 145 kB 00:00 2024-03-25T14:18:00.432 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:00.432 INFO:teuthology.orchestra.run.smithi121.stdout:Total 640 kB/s | 145 kB 00:00 2024-03-25T14:18:00.432 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:00.440 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:00.440 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:00.491 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:00.491 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:00.708 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:18:00.717 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:18:00.718 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:00.718 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:00.718 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:00.718 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:00.718 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:00.731 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:18:00.732 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:18:00.732 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T14:18:00.732 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:00.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T14:18:01.482 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:04 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:01.636 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:01.637 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:18:01.637 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:01.637 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T14:18:01.637 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 145 k 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 436 k 2024-03-25T14:18:01.638 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:18:01.734 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:01.735 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:01.735 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:01.735 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:01.844 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 705 kB/s | 145 kB 00:00 2024-03-25T14:18:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:Total 700 kB/s | 145 kB 00:00 2024-03-25T14:18:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:01.853 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:01.853 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:01.906 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:01.907 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:01.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-03-25T14:18:02.175 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:02.270 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:18:02.294 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:18:02.294 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:18:02.294 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T14:18:02.294 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:02.541 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:02.640 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:02.692 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:02.693 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:02.693 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:02.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T14:18:03.252 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:18:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:03.253 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:03.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:03.253 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:03.320 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:20 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:03.471 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T14:18:03.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-25T14:18:03.473 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-03-25T14:18:03.474 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:03.474 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.9 M 2024-03-25T14:18:03.474 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-03-25T14:18:03.474 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:03.489 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-03-25T14:18:04.062 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:07 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:04.077 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 123 kB/s | 31 kB 00:00 2024-03-25T14:18:04.094 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 56 kB/s | 15 kB 00:00 2024-03-25T14:18:04.162 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:04.218 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:04.220 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:04.220 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:04.245 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T14:18:04.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T14:18:04.303 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T14:18:04.429 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.9 MB/s | 1.8 MB 00:00 2024-03-25T14:18:04.479 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-25T14:18:04.513 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-03-25T14:18:04.621 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-25T14:18:04.663 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 738 kB/s | 31 kB 00:00 2024-03-25T14:18:04.688 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-25T14:18:04.722 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-25T14:18:04.764 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.2 MB/s | 2.0 MB 00:00 2024-03-25T14:18:04.767 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:04.767 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.3 MB/s | 6.9 MB 00:01 2024-03-25T14:18:04.785 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:04.801 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:04.801 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:04.865 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:07 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:04.969 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:04.969 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:05.020 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:05.021 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-03-25T14:18:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:18:05.021 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-03-25T14:18:05.021 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:05.021 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T14:18:05.021 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:05.022 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-03-25T14:18:05.023 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-03-25T14:18:05.023 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:05.023 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.9 M 2024-03-25T14:18:05.023 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-03-25T14:18:05.023 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:05.414 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 161 kB/s | 31 kB 00:00 2024-03-25T14:18:05.440 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 70 kB/s | 15 kB 00:00 2024-03-25T14:18:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T14:18:05.581 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T14:18:05.600 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T14:18:05.646 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T14:18:05.685 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T14:18:05.789 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T14:18:05.807 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.0 MB/s | 1.8 MB 00:00 2024-03-25T14:18:05.874 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-03-25T14:18:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T14:18:06.012 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T14:18:06.056 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T14:18:06.085 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:18:06.116 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 89 kB/s | 19 kB 00:00 2024-03-25T14:18:06.116 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T14:18:06.249 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T14:18:06.316 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-25T14:18:06.359 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T14:18:06.383 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 462 kB/s | 31 kB 00:00 2024-03-25T14:18:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:18:06.442 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.3 MB/s | 2.0 MB 00:00 2024-03-25T14:18:06.467 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T14:18:07.258 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T14:18:07.259 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T14:18:07.400 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 520 kB/s | 1.0 MB 00:01 2024-03-25T14:18:07.406 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:07.407 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.9 MB/s | 6.9 MB 00:02 2024-03-25T14:18:07.427 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:07.447 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:07.448 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T14:18:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:07.609 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:07.624 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:07.624 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:07.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T14:18:08.142 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:08.226 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T14:18:08.261 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T14:18:08.309 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T14:18:08.338 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T14:18:08.419 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:08.438 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T14:18:08.570 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:08.570 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T14:18:08.571 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-03-25T14:18:08.572 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:08.573 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 37 M 2024-03-25T14:18:08.573 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 209 M 2024-03-25T14:18:08.573 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T14:18:08.719 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T14:18:08.746 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:18:08.778 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T14:18:08.863 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-25T14:18:08.905 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 425 kB/s | 17 kB 00:00 2024-03-25T14:18:08.966 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-25T14:18:08.989 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 418 kB/s | 34 kB 00:00 2024-03-25T14:18:09.034 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T14:18:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:18:09.164 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-25T14:18:09.232 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-25T14:18:09.257 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T14:18:09.599 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T14:18:09.791 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T14:18:09.816 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 906 kB/s | 195 kB 00:00 2024-03-25T14:18:09.891 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-25T14:18:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:18:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T14:18:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T14:18:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T14:18:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T14:18:09.931 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T14:18:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T14:18:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T14:18:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T14:18:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:18:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T14:18:09.932 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T14:18:10.199 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.0 MB/s | 7.4 MB 00:01 2024-03-25T14:18:10.203 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:10.203 INFO:teuthology.orchestra.run.smithi121.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-25T14:18:10.242 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:10.265 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:10.276 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T14:18:10.277 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:10.277 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:10.277 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T14:18:10.277 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:10.277 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:10.278 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:10.278 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T14:18:10.278 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T14:18:10.278 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T14:18:10.278 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T14:18:10.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T14:18:10.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T14:18:10.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T14:18:10.279 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T14:18:10.279 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:10.280 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:10.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T14:18:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:10.631 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:11.101 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:14 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:11.257 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T14:18:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-03-25T14:18:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:11.260 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 37 M 2024-03-25T14:18:11.260 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 209 M 2024-03-25T14:18:11.260 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:11.347 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:11.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T14:18:11.453 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T14:18:11.554 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:18:11.598 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T14:18:11.776 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:18:11.798 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 247 kB/s | 32 kB 00:00 2024-03-25T14:18:11.806 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:18:11.831 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-25T14:18:11.884 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T14:18:11.898 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-25T14:18:11.924 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 378 kB/s | 34 kB 00:00 2024-03-25T14:18:12.091 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T14:18:12.149 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-25T14:18:12.174 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T14:18:12.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T14:18:12.525 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T14:18:12.634 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T14:18:12.692 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.2 MB/s | 7.4 MB 00:01 2024-03-25T14:18:12.734 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-25T14:18:12.784 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 759 kB/s | 195 kB 00:00 2024-03-25T14:18:12.850 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-25T14:18:12.854 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:12.855 INFO:teuthology.orchestra.run.smithi026.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-25T14:18:12.895 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:12.919 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:12.919 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:13.307 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:13.307 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T14:18:14.026 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:14.108 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T14:18:14.147 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T14:18:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:18:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T14:18:14.470 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:18:14.507 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:18:14.590 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T14:18:14.867 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T14:18:14.897 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:18:15.274 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T14:18:15.397 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T14:18:16.694 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T14:18:17.399 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T14:18:17.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T14:18:17.761 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T14:18:17.789 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:18:17.817 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T14:18:17.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:17.817 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:17.817 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:17.817 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T14:18:17.818 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T14:18:17.818 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T14:18:17.818 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T14:18:17.818 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T14:18:17.818 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T14:18:17.819 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T14:18:17.819 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T14:18:17.819 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T14:18:17.819 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T14:18:17.819 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T14:18:17.820 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:17.820 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:18.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-03-25T14:18:18.773 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:18.922 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T14:18:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-03-25T14:18:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.0 M 2024-03-25T14:18:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-03-25T14:18:18.926 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:19.114 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 170 kB/s | 24 kB 00:00 2024-03-25T14:18:19.139 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-25T14:18:19.240 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2024-03-25T14:18:19.265 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-25T14:18:19.290 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T14:18:19.307 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T14:18:19.324 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 148 kB/s | 52 kB 00:00 2024-03-25T14:18:19.816 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 29 kB/s | 15 kB 00:00 2024-03-25T14:18:20.067 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 41 kB/s | 32 kB 00:00 2024-03-25T14:18:20.108 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 258 kB/s | 203 kB 00:00 2024-03-25T14:18:20.259 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T14:18:20.281 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T14:18:20.282 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T14:18:20.282 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T14:18:20.292 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-03-25T14:18:20.730 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T14:18:20.731 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T14:18:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T14:18:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T14:18:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T14:18:20.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:20.732 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:21.102 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-03-25T14:18:21.376 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 670 kB/s | 1.0 MB 00:01 2024-03-25T14:18:21.381 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:21.381 INFO:teuthology.orchestra.run.smithi121.stdout:Total 854 kB/s | 2.0 MB 00:02 2024-03-25T14:18:21.381 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:21.396 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:21.396 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:21.492 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:21.493 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:21.681 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:24 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:21.838 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:21.839 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:18:21.839 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:18:21.839 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:18:21.839 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T14:18:21.840 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-03-25T14:18:21.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:21.842 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.0 M 2024-03-25T14:18:21.842 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-03-25T14:18:21.842 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T14:18:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T14:18:22.265 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 111 kB/s | 24 kB 00:00 2024-03-25T14:18:22.290 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 74 kB/s | 18 kB 00:00 2024-03-25T14:18:22.297 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T14:18:22.327 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T14:18:22.369 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T14:18:22.415 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-03-25T14:18:22.440 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 879 kB/s | 132 kB 00:00 2024-03-25T14:18:22.466 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T14:18:22.470 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T14:18:22.491 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-25T14:18:22.516 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 111 kB/s | 52 kB 00:00 2024-03-25T14:18:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T14:18:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T14:18:22.950 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 66 kB/s | 32 kB 00:00 2024-03-25T14:18:22.975 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2024-03-25T14:18:23.159 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-25T14:18:23.309 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 256 kB/s | 203 kB 00:00 2024-03-25T14:18:23.334 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-25T14:18:23.353 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T14:18:23.401 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T14:18:23.434 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T14:18:23.483 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T14:18:23.505 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:18:24.341 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:18:24.341 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T14:18:24.342 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T14:18:24.343 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T14:18:24.343 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T14:18:24.343 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 751 kB/s | 1.0 MB 00:01 2024-03-25T14:18:24.348 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:24.348 INFO:teuthology.orchestra.run.smithi026.stdout:Total 837 kB/s | 2.0 MB 00:02 2024-03-25T14:18:24.349 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:24.365 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:24.365 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:24.471 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:24.471 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T14:18:24.695 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T14:18:24.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T14:18:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T14:18:24.697 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:24.697 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:24.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T14:18:25.160 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T14:18:25.404 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T14:18:25.468 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T14:18:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T14:18:25.534 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:25.545 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T14:18:25.644 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T14:18:25.684 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:25.684 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T14:18:25.685 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T14:18:25.686 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:25.686 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:25.686 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:18:25.686 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-03-25T14:18:25.686 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:25.687 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.8 M 2024-03-25T14:18:25.687 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-03-25T14:18:25.687 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:25.752 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T14:18:25.806 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T14:18:26.019 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-25T14:18:26.086 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 440 kB/s | 138 kB 00:00 2024-03-25T14:18:26.119 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 714 kB/s | 248 kB 00:00 2024-03-25T14:18:26.579 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T14:18:26.626 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T14:18:26.628 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 106 kB/s | 58 kB 00:00 2024-03-25T14:18:26.660 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T14:18:26.670 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 97 kB/s | 51 kB 00:00 2024-03-25T14:18:26.709 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T14:18:26.738 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:18:27.229 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 453 kB/s | 548 kB 00:01 2024-03-25T14:18:27.233 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:27.234 INFO:teuthology.orchestra.run.smithi121.stdout:Total 4.4 MB/s | 6.8 MB 00:01 2024-03-25T14:18:27.247 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:27.258 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:27.259 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:27.609 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T14:18:27.610 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T14:18:27.611 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T14:18:27.611 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T14:18:27.611 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T14:18:27.650 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T14:18:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T14:18:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T14:18:27.964 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T14:18:27.965 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T14:18:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T14:18:27.966 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:27.966 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:28.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T14:18:28.376 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T14:18:28.457 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T14:18:28.519 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T14:18:28.542 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:18:28.794 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:31 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:28.953 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:28.953 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:18:28.953 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:18:28.954 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-25T14:18:28.955 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:28.955 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.8 M 2024-03-25T14:18:28.955 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-03-25T14:18:28.955 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:29.360 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T14:18:29.369 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:18:29.370 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T14:18:29.370 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T14:18:29.370 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T14:18:29.370 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T14:18:29.370 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T14:18:29.477 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 644 kB/s | 248 kB 00:00 2024-03-25T14:18:29.502 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 336 kB/s | 138 kB 00:00 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T14:18:29.670 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T14:18:29.671 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T14:18:29.671 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T14:18:29.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:29.671 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:29.677 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T14:18:29.694 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-03-25T14:18:29.711 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 244 kB/s | 51 kB 00:00 2024-03-25T14:18:29.714 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:29.714 INFO:teuthology.orchestra.run.smithi026.stdout:Total 8.9 MB/s | 6.8 MB 00:00 2024-03-25T14:18:29.727 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:29.738 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:29.738 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:29.830 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:29.830 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:29.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-03-25T14:18:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:30.293 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T14:18:30.347 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T14:18:30.463 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:30.614 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:30.615 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:30.616 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:30.616 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-25T14:18:30.616 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:30.616 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 964 k 2024-03-25T14:18:30.616 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.8 M 2024-03-25T14:18:30.616 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:30.738 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 642 kB/s | 79 kB 00:00 2024-03-25T14:18:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T14:18:30.985 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T14:18:30.997 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 885 kB 00:00 2024-03-25T14:18:30.998 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:30.999 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.5 MB/s | 964 kB 00:00 2024-03-25T14:18:30.999 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:31.019 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:31.019 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:31.054 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T14:18:31.083 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:18:31.091 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:31.092 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:31.279 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T14:18:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:18:31.874 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:18:31.874 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T14:18:31.874 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T14:18:31.874 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T14:18:31.874 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T14:18:31.874 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T14:18:32.169 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:18:32.169 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:32.181 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:32.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-03-25T14:18:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T14:18:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T14:18:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:32.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-03-25T14:18:33.001 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:33.157 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:33.159 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:33.159 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:18:33.159 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:33.159 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T14:18:33.160 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T14:18:33.160 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:33.160 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:33.161 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:33.161 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-25T14:18:33.161 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:33.161 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 964 k 2024-03-25T14:18:33.161 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.8 M 2024-03-25T14:18:33.161 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:33.287 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 616 kB/s | 79 kB 00:00 2024-03-25T14:18:33.336 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:33.493 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:33.494 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:33.495 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-25T14:18:33.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:33.495 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 332 k 2024-03-25T14:18:33.495 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.5 M 2024-03-25T14:18:33.495 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:33.546 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 885 kB 00:00 2024-03-25T14:18:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:33.547 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.4 MB/s | 964 kB 00:00 2024-03-25T14:18:33.548 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:33.569 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:33.610 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 596 kB/s | 69 kB 00:00 2024-03-25T14:18:33.644 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:33.645 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:33.768 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 963 kB/s | 264 kB 00:00 2024-03-25T14:18:33.770 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:33.770 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-25T14:18:33.770 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:33.778 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:33.778 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:33.815 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:33.815 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:33.966 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:33.976 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T14:18:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:18:34.176 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T14:18:34.215 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:18:34.230 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:18:34.230 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:18:34.230 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T14:18:34.230 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:18:34.790 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:18:34.976 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T14:18:35.075 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T14:18:35.075 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:35.075 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:35.075 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T14:18:35.075 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:35.075 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T14:18:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:35.294 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T14:18:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:35.294 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:35.317 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-volume 2024-03-25T14:18:35.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-03-25T14:18:35.901 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:38 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:36.057 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:36.058 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:36.059 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:36.059 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-25T14:18:36.059 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:36.059 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 332 k 2024-03-25T14:18:36.059 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.5 M 2024-03-25T14:18:36.059 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:36.185 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:53 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:36.192 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 516 kB/s | 69 kB 00:00 2024-03-25T14:18:36.335 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:36.336 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:36.337 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:36.337 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:18:36.337 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:36.337 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-03-25T14:18:36.337 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-03-25T14:18:36.337 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:36.451 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 673 kB/s | 264 kB 00:00 2024-03-25T14:18:36.452 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:36.452 INFO:teuthology.orchestra.run.smithi026.stdout:Total 844 kB/s | 332 kB 00:00 2024-03-25T14:18:36.453 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:36.462 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:36.462 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:36.507 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:36.508 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:36.551 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 591 kB/s | 127 kB 00:00 2024-03-25T14:18:36.552 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:36.552 INFO:teuthology.orchestra.run.smithi121.stdout:Total 586 kB/s | 127 kB 00:00 2024-03-25T14:18:36.552 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:36.558 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:36.558 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:36.617 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:36.617 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:36.663 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:36.841 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T14:18:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:36.878 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:18:36.894 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:18:36.895 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:18:36.895 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T14:18:36.895 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:37.361 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:37.661 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:37.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:37.661 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:37.661 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:37.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:37.662 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:37.666 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T14:18:37.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-03-25T14:18:37.989 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T14:18:37.989 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:37.989 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:37.989 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:18:37.989 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T14:18:37.989 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:37.990 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:38.338 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-03-25T14:18:38.445 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:38.547 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:38.602 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:38.603 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:38.603 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:38.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-03-25T14:18:38.923 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:41 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:39.079 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:39.080 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 127 k 2024-03-25T14:18:39.081 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 456 k 2024-03-25T14:18:39.081 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:39.237 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:56 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:39.388 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:39.389 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:39.389 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:18:39.389 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:39.389 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:39.389 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 140 k 2024-03-25T14:18:39.390 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:39.423 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 370 kB/s | 127 kB 00:00 2024-03-25T14:18:39.424 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:39.424 INFO:teuthology.orchestra.run.smithi026.stdout:Total 368 kB/s | 127 kB 00:00 2024-03-25T14:18:39.424 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:39.430 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:39.430 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:39.488 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:39.489 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:39.630 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 130 kB/s | 31 kB 00:00 2024-03-25T14:18:39.631 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:39.631 INFO:teuthology.orchestra.run.smithi121.stdout:Total 129 kB/s | 31 kB 00:00 2024-03-25T14:18:39.631 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:39.635 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:39.635 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:39.641 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:39.663 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:39.666 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:40.209 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:40.503 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:40.504 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:40.504 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:40.504 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:40.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-03-25T14:18:40.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-03-25T14:18:41.260 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:41.304 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:44 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:41.357 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:41.404 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:41.408 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:41.409 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:41.409 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:41.460 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:41.461 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:41.461 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:41.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-03-25T14:18:41.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-03-25T14:18:42.024 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:42.111 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:42.120 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:42.171 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:42.172 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:42.172 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:42.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-03-25T14:18:42.267 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:42.267 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:18:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:42.269 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 31 k 2024-03-25T14:18:42.269 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 140 k 2024-03-25T14:18:42.269 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:42.434 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 188 kB/s | 31 kB 00:00 2024-03-25T14:18:42.435 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:42.435 INFO:teuthology.orchestra.run.smithi026.stdout:Total 186 kB/s | 31 kB 00:00 2024-03-25T14:18:42.435 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:42.439 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:42.439 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:42.466 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:42.466 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:42.600 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:42.711 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:42.786 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:42.882 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:42.935 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:42.936 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:42.936 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:42.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-03-25T14:18:43.078 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:43.337 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:43.337 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:43.337 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:43.337 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:43.337 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:43.337 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:43.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-03-25T14:18:43.551 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:43.647 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:43.700 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:43.701 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:43.701 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:43.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-03-25T14:18:44.091 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:47 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:44.192 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:44.246 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:44.247 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:44.248 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:44.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-03-25T14:18:44.323 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:01 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:44.420 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:44.472 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:44.473 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:44.473 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:44.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-03-25T14:18:44.882 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:47 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:44.982 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:45.037 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:45.038 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:45.038 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:45.082 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:45.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-03-25T14:18:45.180 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:45.232 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:45.233 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:18:45.233 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:45.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-03-25T14:18:45.664 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:45.763 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:45.818 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:45.819 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:45.819 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:45.849 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:45.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-03-25T14:18:45.998 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-25T14:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:45.999 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-03-25T14:18:46.000 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:46.308 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 277 kB/s | 85 kB 00:00 2024-03-25T14:18:46.308 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:46.309 INFO:teuthology.orchestra.run.smithi121.stdout:Total 276 kB/s | 85 kB 00:00 2024-03-25T14:18:46.309 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:46.316 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:46.316 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:46.372 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:46.372 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:46.454 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:46.506 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:46.553 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:46.608 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:46.609 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:46.609 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:46.700 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-03-25T14:18:47.002 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:47.276 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:47.290 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:47.290 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:47.290 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:47.290 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:47.290 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:47.375 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:47.430 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:47.431 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:47.431 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:47.480 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-03-25T14:18:47.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-03-25T14:18:48.054 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:48.124 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:48.153 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:18:48.207 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:48.209 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:18:48.209 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:48.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-03-25T14:18:48.273 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:48.273 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:48.273 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.1 M 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 13 M 2024-03-25T14:18:48.274 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:48.835 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:48.992 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:48.992 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:48.992 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:18:48.993 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:48.994 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 85 k 2024-03-25T14:18:48.994 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 231 k 2024-03-25T14:18:48.994 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:49.144 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.5 MB/s | 3.1 MB 00:00 2024-03-25T14:18:49.145 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:49.145 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.5 MB/s | 3.1 MB 00:00 2024-03-25T14:18:49.146 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:49.154 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:49.154 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:49.200 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 414 kB/s | 85 kB 00:00 2024-03-25T14:18:49.201 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:49.201 INFO:teuthology.orchestra.run.smithi026.stdout:Total 410 kB/s | 85 kB 00:00 2024-03-25T14:18:49.201 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:49.209 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:49.209 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:49.226 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:49.226 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:49.266 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:49.267 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:49.410 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:49.508 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:49.664 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:49.753 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:49.753 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:18:49.753 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T14:18:49.753 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:18:49.753 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:18:49.753 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:49.941 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:50.223 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:50.223 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:50.223 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:50.223 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:50.223 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:50.223 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:50.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-03-25T14:18:50.883 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:50.884 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:50.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:50.884 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:51.003 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:51.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-03-25T14:18:51.160 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.1 M 2024-03-25T14:18:51.161 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 13 M 2024-03-25T14:18:51.162 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:51.676 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:51.734 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.4 MB/s | 3.1 MB 00:00 2024-03-25T14:18:51.734 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:51.735 INFO:teuthology.orchestra.run.smithi026.stdout:Total 5.4 MB/s | 3.1 MB 00:00 2024-03-25T14:18:51.735 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:51.745 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:51.745 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:51.822 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:51.822 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:51.826 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:51.827 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:51.827 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 171 k 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-03-25T14:18:51.828 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:52.273 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:52.335 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:52.348 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 330 kB/s | 171 kB 00:00 2024-03-25T14:18:52.348 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:52.349 INFO:teuthology.orchestra.run.smithi121.stdout:Total 329 kB/s | 171 kB 00:00 2024-03-25T14:18:52.349 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:52.356 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:52.357 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:52.359 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:18:52.359 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T14:18:52.359 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:18:52.359 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:18:52.359 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:52.412 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:52.412 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:18:52.565 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:18:52.662 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:53.082 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:53.357 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:53.357 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:53.357 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:18:53.358 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:53.358 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:53.358 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:18:53.518 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:53.519 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:53.519 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:53.519 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:53.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:18:53.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-03-25T14:18:54.138 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:18:54.292 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:18:54.292 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:54.292 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:18:54.292 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:54.292 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:18:54.292 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-25T14:18:54.293 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:18:54.294 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 91 M 2024-03-25T14:18:54.294 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 498 M 2024-03-25T14:18:54.294 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:18:54.352 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:54.507 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:54.507 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 171 k 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 494 k 2024-03-25T14:18:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:54.853 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 498 kB/s | 171 kB 00:00 2024-03-25T14:18:54.854 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:54.854 INFO:teuthology.orchestra.run.smithi026.stdout:Total 496 kB/s | 171 kB 00:00 2024-03-25T14:18:54.854 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:18:54.865 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:18:54.865 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:18:54.921 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:18:54.921 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:18:55.072 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:18:55.163 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:55.603 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:55.885 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:55.886 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:18:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:55.886 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:55.886 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:18:56.117 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:18:56.694 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:18:56.850 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:18:56.851 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:56.851 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:18:56.851 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:56.851 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-25T14:18:56.852 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:18:56.853 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 91 M 2024-03-25T14:18:56.853 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 498 M 2024-03-25T14:18:56.853 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:18:57.801 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 6.4 MB/s | 22 MB 00:03 2024-03-25T14:18:59.159 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 14 MB/s | 68 MB 00:04 2024-03-25T14:18:59.161 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:59.161 INFO:teuthology.orchestra.run.smithi121.stdout:Total 19 MB/s | 91 MB 00:04 2024-03-25T14:18:59.162 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:18:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:18:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:18:59.753 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:18:59.753 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:19:00.547 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 6.1 MB/s | 22 MB 00:03 2024-03-25T14:19:02.871 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:19:03.388 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 10 MB/s | 68 MB 00:06 2024-03-25T14:19:03.390 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:19:03.391 INFO:teuthology.orchestra.run.smithi026.stdout:Total 14 MB/s | 91 MB 00:06 2024-03-25T14:19:03.391 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:19:03.395 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:19:03.395 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:19:03.999 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:19:04.000 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:19:07.269 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:19:09.980 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:19:10.135 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:19:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:19:13.501 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T14:19:13.768 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:19:13.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:19:13.768 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:19:13.769 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:19:13.769 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:19:13.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:19:13.769 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:19:13.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-common-debuginfo 2024-03-25T14:19:14.529 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:31 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:19:14.678 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 416 M 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.7 G 2024-03-25T14:19:14.679 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:19:14.795 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:19:14.876 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:19:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:19:18.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:19:18.548 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:19:18.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-common-debuginfo 2024-03-25T14:19:19.347 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:22 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:19:19.507 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:19:19.508 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:19:19.508 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:19:19.508 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 416 M 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.7 G 2024-03-25T14:19:19.509 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:19:35.077 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 20 MB/s | 416 MB 00:20 2024-03-25T14:19:35.078 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:19:35.078 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 416 MB 00:20 2024-03-25T14:19:35.078 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:19:35.082 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:19:35.082 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:19:37.551 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:19:37.552 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:19:52.149 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 13 MB/s | 416 MB 00:32 2024-03-25T14:19:52.149 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:19:52.150 INFO:teuthology.orchestra.run.smithi026.stdout:Total 13 MB/s | 416 MB 00:32 2024-03-25T14:19:52.150 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:19:52.154 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:19:52.154 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:19:54.742 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:19:54.743 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:20:19.762 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:20:20.213 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:20:25.765 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:20:26.044 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:20:26.044 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:26.044 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:20:26.044 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:20:26.044 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:26.044 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:20:26.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T14:20:26.857 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:20:27.005 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:20:27.005 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.4 M 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 6.4 M 2024-03-25T14:20:27.006 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:20:27.848 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 1.7 MB/s | 1.4 MB 00:00 2024-03-25T14:20:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:20:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.7 MB/s | 1.4 MB 00:00 2024-03-25T14:20:27.849 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:20:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:20:27.853 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:20:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:20:27.895 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:20:28.147 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:20:28.246 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:20:28.674 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:20:28.952 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:20:28.953 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:28.953 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:20:28.953 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:20:28.953 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:28.953 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:20:29.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T14:20:29.707 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:03:46 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:20:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:20:29.854 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 176 M 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 753 M 2024-03-25T14:20:29.855 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:20:39.358 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:20:39.446 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:20:40.641 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 16 MB/s | 176 MB 00:10 2024-03-25T14:20:40.642 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:20:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:Total 16 MB/s | 176 MB 00:10 2024-03-25T14:20:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:20:40.647 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:20:40.647 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:20:41.705 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:20:41.705 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:20:47.491 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:20:47.743 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:20:47.743 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:47.743 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:20:47.743 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:20:47.744 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:47.744 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:20:47.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T14:20:48.575 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:03:51 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:20:48.731 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 1.4 M 2024-03-25T14:20:48.732 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 6.4 M 2024-03-25T14:20:48.733 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:20:49.346 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 2.3 MB/s | 1.4 MB 00:00 2024-03-25T14:20:49.347 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:20:49.347 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.3 MB/s | 1.4 MB 00:00 2024-03-25T14:20:49.347 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:20:49.351 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:20:49.351 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:20:49.395 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:20:49.396 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:20:49.662 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:20:49.773 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:20:50.190 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:20:50.463 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:20:50.463 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:50.463 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:20:50.463 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:20:50.463 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:50.463 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:20:50.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T14:20:51.230 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:20:51.385 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:20:51.386 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 176 M 2024-03-25T14:20:51.387 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 753 M 2024-03-25T14:20:51.387 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:20:59.969 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:21:00.070 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:21:02.556 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 16 MB/s | 176 MB 00:11 2024-03-25T14:21:02.557 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:21:02.557 INFO:teuthology.orchestra.run.smithi026.stdout:Total 16 MB/s | 176 MB 00:11 2024-03-25T14:21:02.558 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:21:02.563 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:21:02.563 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:21:03.673 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:21:03.673 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:21:04.747 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:21:04.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:21:04.993 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:21:04.993 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:21:04.993 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:21:04.993 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:21:04.994 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:21:05.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test-debuginfo 2024-03-25T14:21:05.770 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:21:05.917 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:21:05.918 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:21:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:21:05.918 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:21:05.918 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 672 M 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 3.1 G 2024-03-25T14:21:05.919 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:21:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:21:22.830 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:21:27.591 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:21:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:21:27.864 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:21:27.864 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:21:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:21:27.865 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:21:27.865 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:21:28.070 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test-debuginfo 2024-03-25T14:21:28.645 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:04:31 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:21:28.800 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:21:28.801 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 672 M 2024-03-25T14:21:28.802 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 3.1 G 2024-03-25T14:21:28.802 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:21:37.111 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 22 MB/s | 672 MB 00:31 2024-03-25T14:21:37.111 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:21:37.112 INFO:teuthology.orchestra.run.smithi121.stdout:Total 22 MB/s | 672 MB 00:31 2024-03-25T14:21:37.112 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:21:37.117 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:21:37.118 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:21:41.072 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:21:41.072 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:21:59.645 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 22 MB/s | 672 MB 00:30 2024-03-25T14:21:59.647 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:21:59.647 INFO:teuthology.orchestra.run.smithi026.stdout:Total 22 MB/s | 672 MB 00:30 2024-03-25T14:21:59.647 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:21:59.652 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:21:59.652 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:22:03.803 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:22:03.804 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:22:51.894 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:22:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:23:01.586 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:23:01.842 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:23:01.842 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:01.842 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:23:01.842 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:01.842 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:01.843 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:23:02.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:23:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:23:02.808 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:23:02.859 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:23:02.860 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-25T14:23:02.860 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:23:02.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T14:23:03.487 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:23:03.636 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:23:03.637 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:03.638 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 20 M 2024-03-25T14:23:03.638 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 103 M 2024-03-25T14:23:03.638 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:23:04.943 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 15 MB/s | 20 MB 00:01 2024-03-25T14:23:04.943 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:04.944 INFO:teuthology.orchestra.run.smithi121.stdout:Total 15 MB/s | 20 MB 00:01 2024-03-25T14:23:04.944 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:23:04.948 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:23:04.948 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:23:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:23:05.097 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:23:07.333 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:23:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:08.541 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:08.812 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:08.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:08.812 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:23:08.812 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:08.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:08.812 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:23:08.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T14:23:09.553 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:23:09.700 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:23:09.702 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:09.702 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:23:09.702 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:09.702 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 48 M 2024-03-25T14:23:09.702 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 173 M 2024-03-25T14:23:09.702 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:23:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 20 MB/s | 48 MB 00:02 2024-03-25T14:23:12.104 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-25T14:23:12.105 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:23:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:23:12.109 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:23:12.421 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:23:12.421 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:23:17.157 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:23:17.277 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:18.676 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:18.679 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:23:18.789 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:23:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:18.939 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:23:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:18.940 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:23:19.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T14:23:19.698 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:23:19.845 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:23:19.846 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:19.847 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 99 M 2024-03-25T14:23:19.847 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 449 M 2024-03-25T14:23:19.847 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:23:25.955 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 16 MB/s | 99 MB 00:06 2024-03-25T14:23:25.955 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:25.956 INFO:teuthology.orchestra.run.smithi121.stdout:Total 16 MB/s | 99 MB 00:06 2024-03-25T14:23:25.956 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:23:25.960 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:23:25.960 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:23:26.568 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:23:26.569 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:23:27.615 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:23:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:23:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:27.876 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:23:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:27.876 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:27.876 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:23:28.137 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:23:28.712 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:06:31 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:23:28.814 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:23:28.871 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:23:28.872 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-25T14:23:28.873 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:23:28.941 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T14:23:29.513 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:06:32 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:23:29.667 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:23:29.668 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:29.669 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:23:29.669 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:29.669 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 20 M 2024-03-25T14:23:29.669 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 103 M 2024-03-25T14:23:29.669 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:23:30.940 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 16 MB/s | 20 MB 00:01 2024-03-25T14:23:30.941 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:30.941 INFO:teuthology.orchestra.run.smithi026.stdout:Total 15 MB/s | 20 MB 00:01 2024-03-25T14:23:30.941 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:23:30.945 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:23:30.945 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:23:31.100 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:23:31.100 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:23:33.460 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:23:33.575 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:34.532 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:34.799 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:34.800 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:23:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:34.800 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:34.800 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:23:34.973 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T14:23:35.550 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:06:38 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:23:35.703 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:23:35.704 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:35.704 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:23:35.704 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:35.704 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 48 M 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 173 M 2024-03-25T14:23:35.705 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:23:36.846 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:23:36.944 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:37.984 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 21 MB/s | 48 MB 00:02 2024-03-25T14:23:37.984 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:37.985 INFO:teuthology.orchestra.run.smithi026.stdout:Total 21 MB/s | 48 MB 00:02 2024-03-25T14:23:37.985 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:23:37.989 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:23:37.989 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:23:38.318 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:23:38.318 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:23:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:40.142 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:23:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:40.142 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:23:40.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T14:23:40.880 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:06:57 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:23:41.027 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:23:41.028 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 287 M 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.2 G 2024-03-25T14:23:41.029 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:23:43.463 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:23:43.560 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:44.903 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:45.175 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:23:45.175 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:45.175 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:23:45.175 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:23:45.175 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:45.175 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:23:45.363 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T14:23:45.940 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:23:46.097 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:23:46.098 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:23:46.098 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:23:46.098 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 99 M 2024-03-25T14:23:46.098 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 449 M 2024-03-25T14:23:46.098 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:23:55.335 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 99 MB 00:09 2024-03-25T14:23:55.336 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:55.336 INFO:teuthology.orchestra.run.smithi026.stdout:Total 11 MB/s | 99 MB 00:09 2024-03-25T14:23:55.336 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:23:55.341 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:23:55.341 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:23:55.786 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 19 MB/s | 287 MB 00:14 2024-03-25T14:23:55.787 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:23:55.787 INFO:teuthology.orchestra.run.smithi121.stdout:Total 19 MB/s | 287 MB 00:14 2024-03-25T14:23:55.788 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:23:55.792 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:23:55.792 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:23:55.979 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:23:55.980 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:23:57.506 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:23:57.507 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:24:06.858 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:24:06.952 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:09.865 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:10.137 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:10.137 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:24:10.137 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:24:10.137 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:24:10.137 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:24:10.137 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:24:10.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T14:24:10.894 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:07:13 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:24:11.050 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:24:11.050 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:24:11.050 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:24:11.050 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 287 M 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.2 G 2024-03-25T14:24:11.051 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:24:23.522 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 23 MB/s | 287 MB 00:12 2024-03-25T14:24:23.522 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:24:23.522 INFO:teuthology.orchestra.run.smithi026.stdout:Total 23 MB/s | 287 MB 00:12 2024-03-25T14:24:23.523 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:24:23.523 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:24:23.524 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:24:25.315 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:24:25.315 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:24:26.911 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:24:27.010 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:34.608 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:34.884 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:24:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:24:34.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:34.885 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:24:35.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T14:24:35.648 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:07:52 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:24:35.797 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:24:35.798 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 5.9 M 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 28 M 2024-03-25T14:24:35.799 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 12 MB/s | 5.9 MB 00:00 2024-03-25T14:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout:Total 12 MB/s | 5.9 MB 00:00 2024-03-25T14:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:24:36.381 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:24:36.381 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:24:37.128 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:24:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:24:37.810 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:24:38.065 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:24:38.065 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:38.066 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:24:38.066 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:24:38.066 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:38.066 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:24:38.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel-debuginfo 2024-03-25T14:24:38.807 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:07:55 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:24:38.956 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:24:38.956 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 109 k 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 440 k 2024-03-25T14:24:38.957 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:24:39.178 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 493 kB/s | 109 kB 00:00 2024-03-25T14:24:39.180 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:24:39.180 INFO:teuthology.orchestra.run.smithi121.stdout:Total 487 kB/s | 109 kB 00:00 2024-03-25T14:24:39.180 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:24:39.184 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:24:39.184 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:24:39.218 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:24:39.218 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:24:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:24:39.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:24:39.801 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:24:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:24:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:40.072 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:24:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:24:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:40.072 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:24:40.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T14:24:40.820 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:07:57 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:24:40.967 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:40.968 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:24:40.969 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:40.969 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:24:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:40.969 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 5.4 M 2024-03-25T14:24:40.969 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 25 M 2024-03-25T14:24:40.969 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:24:41.431 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 12 MB/s | 5.4 MB 00:00 2024-03-25T14:24:41.432 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:24:41.432 INFO:teuthology.orchestra.run.smithi121.stdout:Total 12 MB/s | 5.4 MB 00:00 2024-03-25T14:24:41.433 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:24:41.436 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:24:41.436 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:24:41.500 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:24:41.500 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:24:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:24:42.288 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:24:42.814 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:24:43.097 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:24:43.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:43.097 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:24:43.097 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:24:43.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:43.097 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:24:43.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2-debuginfo 2024-03-25T14:24:43.854 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:00 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:24:44.003 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:24:44.003 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:44.003 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 44 M 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 253 M 2024-03-25T14:24:44.004 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:24:46.334 INFO:teuthology.orchestra.run.smithi121.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 19 MB/s | 44 MB 00:02 2024-03-25T14:24:46.334 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:24:46.335 INFO:teuthology.orchestra.run.smithi121.stdout:Total 19 MB/s | 44 MB 00:02 2024-03-25T14:24:46.335 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:24:46.339 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:24:46.339 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:24:46.638 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:24:46.639 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:24:51.701 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:24:51.798 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:24:53.621 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:24:53.877 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:24:53.877 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:53.877 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:24:53.877 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:24:53.877 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:53.878 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:24:54.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1-debuginfo 2024-03-25T14:24:54.629 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:11 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:24:54.777 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:24:54.778 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:24:54.778 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 52 M 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 308 M 2024-03-25T14:24:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:24:56.674 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:24:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:24:57.244 INFO:teuthology.orchestra.run.smithi121.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 21 MB/s | 52 MB 00:02 2024-03-25T14:24:57.245 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:24:57.246 INFO:teuthology.orchestra.run.smithi121.stdout:Total 21 MB/s | 52 MB 00:02 2024-03-25T14:24:57.246 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:24:57.250 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:24:57.250 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:24:57.594 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:24:57.594 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:03.576 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:04.499 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:04.804 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:04.805 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:05.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T14:25:05.595 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:08 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:05.751 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:05.751 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:05.751 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 5.9 M 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 28 M 2024-03-25T14:25:05.752 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:05.872 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:06.142 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:06.142 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:06.142 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:06.142 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:06.143 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:06.143 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:06.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T14:25:06.913 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:23 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:06.959 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 4.9 MB/s | 5.9 MB 00:01 2024-03-25T14:25:06.960 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:06.960 INFO:teuthology.orchestra.run.smithi026.stdout:Total 4.9 MB/s | 5.9 MB 00:01 2024-03-25T14:25:06.960 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:06.964 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:06.964 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:07.037 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:07.038 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:07.061 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:07.062 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:07.062 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:07.062 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:07.062 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:07.062 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 615 k 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.1 M 2024-03-25T14:25:07.063 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:07.496 INFO:teuthology.orchestra.run.smithi121.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 1.4 MB/s | 615 kB 00:00 2024-03-25T14:25:07.497 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:07.497 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.4 MB/s | 615 kB 00:00 2024-03-25T14:25:07.497 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:07.537 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:07.537 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:07.700 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:07.795 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:07.810 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:07.903 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:25:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:08.458 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:08.458 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:08.458 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:08.458 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:08.458 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:08.458 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:08.468 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:25:08.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados-debuginfo 2024-03-25T14:25:08.737 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:25:08.737 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:08.737 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:08.737 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:08.737 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:08.737 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:08.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel-debuginfo 2024-03-25T14:25:09.220 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:26 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:09.370 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:09.370 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 771 k 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.7 M 2024-03-25T14:25:09.371 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:09.503 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:12 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:09.659 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:09.660 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:09.661 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 109 k 2024-03-25T14:25:09.661 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 440 k 2024-03-25T14:25:09.661 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:09.674 INFO:teuthology.orchestra.run.smithi121.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 2.5 MB/s | 771 kB 00:00 2024-03-25T14:25:09.675 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:09.675 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.5 MB/s | 771 kB 00:00 2024-03-25T14:25:09.675 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:09.678 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:09.679 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:09.717 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:09.717 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:09.917 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:10.033 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:25:10.162 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 217 kB/s | 109 kB 00:00 2024-03-25T14:25:10.162 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:10.162 INFO:teuthology.orchestra.run.smithi026.stdout:Total 217 kB/s | 109 kB 00:00 2024-03-25T14:25:10.163 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:10.166 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:10.166 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:10.202 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:10.202 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:10.322 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:10.431 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:10.456 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:25:10.716 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:25:10.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:10.716 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:10.717 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:10.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:10.717 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:10.797 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:10.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T14:25:11.060 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:11.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:11.060 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:11.060 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:11.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:11.060 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:11.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T14:25:11.508 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:28 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:11.657 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:11.658 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:11.659 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.1 M 2024-03-25T14:25:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 4.3 M 2024-03-25T14:25:11.659 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:11.827 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:14 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:11.981 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:11.982 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:11.982 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:11.982 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:11.982 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 5.4 M 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 25 M 2024-03-25T14:25:11.983 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:12.178 INFO:teuthology.orchestra.run.smithi121.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 2.2 MB/s | 1.1 MB 00:00 2024-03-25T14:25:12.178 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:12.178 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.2 MB/s | 1.1 MB 00:00 2024-03-25T14:25:12.178 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:12.182 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:12.221 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:12.221 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:12.437 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:12.544 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:12.606 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 8.6 MB/s | 5.4 MB 00:00 2024-03-25T14:25:12.607 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:12.607 INFO:teuthology.orchestra.run.smithi026.stdout:Total 8.6 MB/s | 5.4 MB 00:00 2024-03-25T14:25:12.607 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:12.611 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:12.611 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:12.678 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:12.679 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:12.970 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:13.224 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:13.224 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:13.224 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:13.224 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:13.224 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:13.224 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:13.374 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:13.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T14:25:13.475 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:25:14.018 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:31 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:14.028 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:25:14.168 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:14.169 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:14.169 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:14.169 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:14.169 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 337 k 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.0 M 2024-03-25T14:25:14.170 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:25:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:14.328 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:14.328 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:14.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2-debuginfo 2024-03-25T14:25:14.556 INFO:teuthology.orchestra.run.smithi121.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 875 kB/s | 337 kB 00:00 2024-03-25T14:25:14.556 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:14.557 INFO:teuthology.orchestra.run.smithi121.stdout:Total 872 kB/s | 337 kB 00:00 2024-03-25T14:25:14.557 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:14.560 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:14.560 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:14.595 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:14.595 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:14.733 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:14.838 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:15.097 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:18 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:15.251 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:15.252 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:15.252 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:15.252 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:15.252 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 44 M 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 253 M 2024-03-25T14:25:15.253 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:15.269 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:15.523 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:15.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T14:25:16.306 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:33 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T14:25:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 509 k 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.5 M 2024-03-25T14:25:16.457 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:16.717 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.9 MB/s | 509 kB 00:00 2024-03-25T14:25:16.718 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:16.718 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.9 MB/s | 509 kB 00:00 2024-03-25T14:25:16.718 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:16.721 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:16.722 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:16.758 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:16.758 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:17.699 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:17.699 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:17.699 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:17.699 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:17.699 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:17.699 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:17.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T14:25:18.342 INFO:teuthology.orchestra.run.smithi026.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 14 MB/s | 44 MB 00:03 2024-03-25T14:25:18.343 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:18.344 INFO:teuthology.orchestra.run.smithi026.stdout:Total 14 MB/s | 44 MB 00:03 2024-03-25T14:25:18.344 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:18.348 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:18.348 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:18.472 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:35 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:18.621 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:18.622 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:18.623 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:18.623 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:18.623 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:18.623 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 52 M 2024-03-25T14:25:18.623 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 298 M 2024-03-25T14:25:18.623 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:18.661 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:18.661 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:22.628 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 13 MB/s | 52 MB 00:04 2024-03-25T14:25:22.629 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:22.629 INFO:teuthology.orchestra.run.smithi121.stdout:Total 13 MB/s | 52 MB 00:04 2024-03-25T14:25:22.629 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:22.633 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:22.633 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:22.969 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:22.969 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:23.996 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:24.102 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:25:25.935 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:25:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:25:26.199 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:26.199 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:26.200 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:26.200 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:26.405 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1-debuginfo 2024-03-25T14:25:26.984 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:29 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:27.143 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:27.144 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:27.144 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 52 M 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 308 M 2024-03-25T14:25:27.145 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:28.855 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:29.108 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:25:29.786 INFO:teuthology.orchestra.run.smithi026.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 20 MB/s | 52 MB 00:02 2024-03-25T14:25:29.787 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:29.788 INFO:teuthology.orchestra.run.smithi026.stdout:Total 20 MB/s | 52 MB 00:02 2024-03-25T14:25:29.788 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:29.792 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:29.792 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:30.152 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:30.152 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:31.152 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:25:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:25:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:31.402 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:31.402 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:31.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T14:25:32.173 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:08:49 ago on Mon 25 Mar 2024 02:16:43 PM UTC. 2024-03-25T14:25:32.322 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-25T14:25:32.323 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:32.323 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-25T14:25:32.323 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:32.323 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-25T14:25:32.323 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.3 M 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 5.5 M 2024-03-25T14:25:32.324 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-25T14:25:32.821 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.5 MB/s | 1.3 MB 00:00 2024-03-25T14:25:32.822 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:32.822 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.5 MB/s | 1.3 MB 00:00 2024-03-25T14:25:32.822 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-25T14:25:32.825 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-25T14:25:32.825 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-25T14:25:32.865 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-25T14:25:32.866 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-25T14:25:33.128 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-25T14:25:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:33.695 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-25T14:25:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:25:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-25T14:25:34.164 DEBUG:teuthology.parallel:result is None 2024-03-25T14:25:36.493 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:36.588 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:38.800 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:39.082 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:25:39.083 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:39.083 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:39.083 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:39.083 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:39.083 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:39.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T14:25:39.858 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:42 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:40.021 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:40.022 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:40.022 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:40.022 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:40.022 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:40.022 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T14:25:40.022 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 615 k 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.1 M 2024-03-25T14:25:40.023 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:40.781 INFO:teuthology.orchestra.run.smithi026.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 812 kB/s | 615 kB 00:00 2024-03-25T14:25:40.781 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:40.782 INFO:teuthology.orchestra.run.smithi026.stdout:Total 810 kB/s | 615 kB 00:00 2024-03-25T14:25:40.782 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:40.786 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:40.786 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:40.828 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:40.829 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:41.003 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:41.107 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:41.494 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:41.764 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:25:41.765 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:41.765 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:41.765 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:42.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados-debuginfo 2024-03-25T14:25:42.596 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:45 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:42.753 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:42.753 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 771 k 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.7 M 2024-03-25T14:25:42.754 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:43.200 INFO:teuthology.orchestra.run.smithi026.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 1.7 MB/s | 771 kB 00:00 2024-03-25T14:25:43.201 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:43.201 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.7 MB/s | 771 kB 00:00 2024-03-25T14:25:43.201 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:43.204 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:43.204 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:43.244 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:43.244 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:43.435 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:43.552 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:25:43.960 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:25:44.222 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:25:44.222 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:44.222 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:44.223 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:44.223 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:44.223 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:44.435 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T14:25:45.005 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:48 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:45.163 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:45.164 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 1.1 M 2024-03-25T14:25:45.165 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 4.3 M 2024-03-25T14:25:45.165 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:45.612 INFO:teuthology.orchestra.run.smithi026.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 2.5 MB/s | 1.1 MB 00:00 2024-03-25T14:25:45.613 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:45.613 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.5 MB/s | 1.1 MB 00:00 2024-03-25T14:25:45.613 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:45.617 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:45.617 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:45.657 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:45.657 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:45.882 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:45.989 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:46.361 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:46.712 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:46.713 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:46.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:46.713 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:47.076 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T14:25:47.648 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:50 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:47.805 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:47.806 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 337 k 2024-03-25T14:25:47.807 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.0 M 2024-03-25T14:25:47.807 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:48.065 INFO:teuthology.orchestra.run.smithi026.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 337 kB 00:00 2024-03-25T14:25:48.066 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:48.066 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-03-25T14:25:48.066 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:48.069 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:48.069 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:48.106 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:48.107 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:48.247 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:48.475 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:48.821 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:49.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:25:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:49.106 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:49.106 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:49.106 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:49.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T14:25:49.894 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:52 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:50.049 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:50.050 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:50.050 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:50.050 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 509 k 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.5 M 2024-03-25T14:25:50.051 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:50.460 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 509 kB 00:00 2024-03-25T14:25:50.460 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:50.460 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.2 MB/s | 509 kB 00:00 2024-03-25T14:25:50.460 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:50.464 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:50.464 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:50.501 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:50.502 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:25:50.671 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:25:50.770 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:51.445 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:25:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:51.446 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:25:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:25:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:51.446 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:25:51.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T14:25:52.238 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:08:55 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:25:52.393 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 52 M 2024-03-25T14:25:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 298 M 2024-03-25T14:25:52.395 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:25:54.880 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 21 MB/s | 52 MB 00:02 2024-03-25T14:25:54.880 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:25:54.881 INFO:teuthology.orchestra.run.smithi026.stdout:Total 21 MB/s | 52 MB 00:02 2024-03-25T14:25:54.881 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:25:54.885 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:25:54.885 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:25:55.237 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:25:55.237 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:26:01.501 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:26:01.606 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:26:03.641 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:26:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:26:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:03.919 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:26:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:26:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:03.919 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:26:04.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T14:26:04.696 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:09:07 ago on Mon 25 Mar 2024 02:16:57 PM UTC. 2024-03-25T14:26:04.850 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-25T14:26:04.851 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:26:04.851 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-25T14:26:04.851 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:26:04.851 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-25T14:26:04.851 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 1.3 M 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 5.5 M 2024-03-25T14:26:04.852 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-25T14:26:05.314 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 2.7 MB/s | 1.3 MB 00:00 2024-03-25T14:26:05.315 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:26:05.315 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-03-25T14:26:05.315 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-25T14:26:05.319 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-25T14:26:05.319 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-25T14:26:05.361 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-25T14:26:05.361 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-25T14:26:05.608 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-25T14:26:05.724 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:26:06.191 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:26:06.468 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:26:06.468 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:06.468 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-25T14:26:06.468 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:26:06.468 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:06.468 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-25T14:26:06.671 DEBUG:teuthology.parallel:result is None 2024-03-25T14:26:06.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:26:06.913 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T14:26:06.933 INFO:teuthology.orchestra.run.smithi026.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T14:26:06.934 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T14:26:06.934 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T14:26:06.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:26:07.165 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T14:26:07.189 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T14:26:07.190 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T14:26:07.190 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T14:26:07.192 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T14:26:07.192 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:07.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T14:26:07.220 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:07.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T14:26:07.260 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T14:26:07.260 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:07.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T14:26:07.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T14:26:07.356 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:07.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T14:26:07.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T14:26:07.452 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T14:26:07.452 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:07.453 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T14:26:07.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T14:26:07.547 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:07.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T14:26:07.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T14:26:07.642 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T14:26:07.643 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:07.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T14:26:07.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T14:26:07.737 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:07.737 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T14:26:07.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T14:26:07.830 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T14:26:07.915 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T14:26:07.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-03-25T14:26:07.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-03-25T14:26:07.944 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T14:26:07.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T14:26:07.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T14:26:08.011 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T14:26:08.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T14:26:08.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T14:26:08.105 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T14:26:08.105 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T14:26:08.105 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621287', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLLiXIgngNtdUUxjOEdjf74oeU6tS2Dqa8Z23Rg++8KRtAagxxs2CR8ZaYaXIpIn8ME7NRK96dK8IlUl2h3juVg=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvUXbyBTSDKsGDyI5JnMlAwPUD+cGxsn5VWUYR714IAgvBMNY3l3MWwkqHEz6JahOYUCIE5jkKSzVxpToE2Kvc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T14:26:08.106 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T14:26:08.119 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T14:26:08.134 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T14:26:08.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T14:26:08.201 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:08.201 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T14:26:08.252 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T14:26:08.252 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-25 14:18:52.606386924 +0000 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-25 14:15:49.258420112 +0000 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-25 14:15:49.258420112 +0000 2024-03-25T14:26:08.307 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-25 14:15:49.258420112 +0000 2024-03-25T14:26:08.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T14:26:08.372 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-25T14:26:08.373 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-25T14:26:08.373 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.7752e-05 s, 5.8 MB/s 2024-03-25T14:26:08.374 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T14:26:08.430 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-03-25T14:26:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-25 14:18:52.607386909 +0000 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-25 14:15:49.590414909 +0000 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-25 14:15:49.590414909 +0000 2024-03-25T14:26:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-25 14:15:49.590414909 +0000 2024-03-25T14:26:08.485 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T14:26:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-25T14:26:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-25T14:26:08.548 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.8152e-05 s, 5.8 MB/s 2024-03-25T14:26:08.549 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T14:26:08.605 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-03-25T14:26:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T14:26:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:08.659 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T14:26:08.659 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:08.659 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:08.659 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-25 14:18:52.607386909 +0000 2024-03-25T14:26:08.660 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-25 14:15:49.927409628 +0000 2024-03-25T14:26:08.660 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-25 14:15:49.927409628 +0000 2024-03-25T14:26:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-25 14:15:49.927409628 +0000 2024-03-25T14:26:08.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T14:26:08.724 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-25T14:26:08.724 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-25T14:26:08.724 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.3864e-05 s, 5.5 MB/s 2024-03-25T14:26:08.725 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T14:26:08.781 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-03-25T14:26:08.834 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T14:26:08.835 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:08.835 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T14:26:08.835 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:08.835 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:08.835 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-25 14:18:52.607386909 +0000 2024-03-25T14:26:08.835 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-25 14:15:50.248404598 +0000 2024-03-25T14:26:08.836 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-25 14:15:50.248404598 +0000 2024-03-25T14:26:08.836 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-25 14:15:50.248404598 +0000 2024-03-25T14:26:08.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T14:26:08.899 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-25T14:26:08.899 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-25T14:26:08.900 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.879e-05 s, 5.2 MB/s 2024-03-25T14:26:08.901 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T14:26:08.956 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:08.956 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:08.956 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T14:26:08.973 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T14:26:08.973 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-03-25T14:26:09.028 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-25 14:18:50.004258041 +0000 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-25 14:15:40.112302968 +0000 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-25 14:15:40.112302968 +0000 2024-03-25T14:26:09.029 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-25 14:15:40.112302968 +0000 2024-03-25T14:26:09.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T14:26:09.095 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-25T14:26:09.095 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-25T14:26:09.095 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000187449 s, 2.7 MB/s 2024-03-25T14:26:09.096 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T14:26:09.153 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-03-25T14:26:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-25 14:18:50.005258027 +0000 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-25 14:15:40.470298078 +0000 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-25 14:15:40.470298078 +0000 2024-03-25T14:26:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-25 14:15:40.470298078 +0000 2024-03-25T14:26:09.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T14:26:09.273 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-25T14:26:09.273 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-25T14:26:09.273 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000181652 s, 2.8 MB/s 2024-03-25T14:26:09.274 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T14:26:09.331 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-25 14:18:50.005258027 +0000 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-25 14:15:40.792293682 +0000 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-25 14:15:40.792293682 +0000 2024-03-25T14:26:09.498 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-25 14:15:40.792293682 +0000 2024-03-25T14:26:09.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T14:26:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-25T14:26:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-25T14:26:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000231181 s, 2.2 MB/s 2024-03-25T14:26:09.564 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T14:26:09.620 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-03-25T14:26:09.675 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T14:26:09.675 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:26:09.675 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T14:26:09.676 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:26:09.676 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:26:09.676 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-25 14:18:50.006258014 +0000 2024-03-25T14:26:09.676 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-25 14:15:41.135288999 +0000 2024-03-25T14:26:09.676 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-25 14:15:41.135288999 +0000 2024-03-25T14:26:09.676 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-25 14:15:41.135288999 +0000 2024-03-25T14:26:09.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T14:26:09.741 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-25T14:26:09.741 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-25T14:26:09.741 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000207172 s, 2.5 MB/s 2024-03-25T14:26:09.742 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T14:26:09.798 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:09.798 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T14:26:09.798 INFO:tasks.ceph:Generating config... 2024-03-25T14:26:09.802 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T14:26:09.802 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T14:26:09.802 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T14:26:09.803 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T14:26:09.803 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T14:26:09.803 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T14:26:09.803 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T14:26:09.803 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T14:26:09.803 INFO:tasks.ceph:[global] ms type = async 2024-03-25T14:26:09.803 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T14:26:09.803 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T14:26:09.804 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T14:26:09.804 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T14:26:09.804 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T14:26:09.804 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T14:26:09.804 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T14:26:09.804 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T14:26:09.805 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T14:26:09.806 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T14:26:09.806 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T14:26:09.806 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T14:26:09.806 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T14:26:09.806 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T14:26:09.806 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T14:26:09.806 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T14:26:09.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T14:26:09.888 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T14:26:09.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T14:26:09.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T14:26:09.967 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.26:6789]'), ('mon.c', '[v1:172.21.15.26:6790]'), ('mon.b', '[v1:172.21.15.121:6789]')] 2024-03-25T14:26:09.968 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.121:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T14:26:09.969 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:09.969 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T14:26:10.022 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.26:6789]' --addv c '[v1:172.21.15.26:6790]' --addv b '[v1:172.21.15.121:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 17 (quincy) 2024-03-25T14:26:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 3 2024-03-25T14:26:10.110 INFO:teuthology.orchestra.run.smithi026.stdout:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:26:10.110 INFO:teuthology.orchestra.run.smithi026.stdout:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:26:10.110 INFO:teuthology.orchestra.run.smithi026.stdout:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:26:10.110 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T14:26:10.111 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T14:26:10.125 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3ca971d3-87b7-4101-bfd9-2dfef6094512... 2024-03-25T14:26:10.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T14:26:10.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T14:26:10.215 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-03-25T14:26:10.216 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T14:26:10.217 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-03-25T14:26:10.218 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.121:6789]" 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: mon client directed command retry = 5 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: mon election default strategy = 3 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: ms bind msgr2 = False 2024-03-25T14:26:10.219 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject socket failures = 5000 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: ms type = async 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd heartbeat grace = 80 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-03-25T14:26:10.220 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug trim objects = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.221 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bdev async discard = True 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bdev enable discard = True 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore allocator = stupid 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore zero block detection = True 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 1 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 1 2024-03-25T14:26:10.222 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd blocked scrub grace period = 3600 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd class default list = * 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd class load list = * 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd client watch timeout = 120 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify cached snaps = True 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify missing on start = True 2024-03-25T14:26:10.223 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd fast shutdown = False 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd max markdown count = 1000 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock override recovery settings = True 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue = debug_random 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue cut off = debug_random 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: osd shutdown pgref assert = True 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:26:10.224 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:26:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-03-25T14:26:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: mon min osdmap epochs = 50 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd crush smoke test = False 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune interval = 2 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune min = 15 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune txsize = 2 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub interval = 300 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on pool no app = False 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: paxos service trim min = 10 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T14:26:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: debug objecter = 20 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: debug rados = 20 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2024-03-25T14:26:10.228 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-03-25T14:26:10.229 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.230 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-03-25T14:26:10.231 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.121:6789]" 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: mon client directed command retry = 5 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 3 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr2 = False 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject socket failures = 5000 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: ms type = async 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: osd heartbeat grace = 80 2024-03-25T14:26:10.232 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.233 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-03-25T14:26:10.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = stupid 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-03-25T14:26:10.235 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1 2024-03-25T14:26:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1 2024-03-25T14:26:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-03-25T14:26:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-25T14:26:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-03-25T14:26:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-03-25T14:26:10.236 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd blocked scrub grace period = 3600 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd class default list = * 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd class load list = * 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd client watch timeout = 120 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-03-25T14:26:10.237 INFO:teuthology.orchestra.run.smithi121.stdout: osd fast shutdown = False 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: osd max markdown count = 1000 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: osd shutdown pgref assert = True 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:26:10.238 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-03-25T14:26:10.239 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T14:26:10.240 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon min osdmap epochs = 50 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd crush smoke test = False 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune interval = 2 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune min = 15 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune txsize = 2 2024-03-25T14:26:10.241 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on pool no app = False 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: paxos service trim min = 10 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-03-25T14:26:10.242 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: debug objecter = 20 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout: debug rados = 20 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-03-25T14:26:10.243 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-03-25T14:26:10.245 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T14:26:10.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T14:26:10.301 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T14:26:10.301 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:10.301 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T14:26:10.315 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:10.315 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T14:26:10.370 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:26:10.370 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:10.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T14:26:10.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T14:26:10.450 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:10.450 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:10.504 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:26:10.504 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:10.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T14:26:10.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T14:26:10.543 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:10.543 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:10.598 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T14:26:10.598 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T14:26:10.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T14:26:10.652 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T14:26:10.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T14:26:10.721 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T14:26:10.724 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T14:26:10.724 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T14:26:10.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T14:26:10.766 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T14:26:10.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T14:26:10.822 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T14:26:10.838 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T14:26:10.838 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T14:26:10.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T14:26:10.864 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:10.865 INFO:tasks.ceph:role: osd.0 2024-03-25T14:26:10.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:26:10.865 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T14:26:10.930 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:10.930 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:10.930 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:10.930 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:10.930 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:10.931 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:10.931 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:10.931 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:10.931 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:10.931 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:10.964 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-25T14:26:10.965 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-25T14:26:10.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T14:26:11.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T14:26:11.107 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T14:26:11.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:11.175 INFO:tasks.ceph:role: osd.1 2024-03-25T14:26:11.175 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:26:11.175 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T14:26:11.240 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:11.241 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:11.241 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:11.241 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:11.241 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:11.273 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-25T14:26:11.274 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-25T14:26:11.275 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T14:26:11.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T14:26:11.416 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T14:26:11.483 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:11.483 INFO:tasks.ceph:role: osd.2 2024-03-25T14:26:11.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:26:11.483 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T14:26:11.549 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:11.549 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:11.550 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:11.581 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-25T14:26:11.583 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-25T14:26:11.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T14:26:11.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T14:26:11.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T14:26:11.789 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:11.789 INFO:tasks.ceph:role: osd.3 2024-03-25T14:26:11.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:26:11.790 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T14:26:11.855 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:11.855 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:11.855 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:11.856 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:11.856 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:11.856 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:11.856 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:11.887 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-25T14:26:11.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-25T14:26:11.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T14:26:11.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T14:26:12.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:12.143+0000 7fc0a1ea8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T14:26:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:12.143+0000 7fc0a1ea8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T14:26:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:12.143+0000 7fc0a1ea8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T14:26:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:12.143+0000 7fc0a1ea8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T14:26:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:12.143+0000 7fc0a1ea8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T14:26:12.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:12.144+0000 7fc0a1ea8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T14:26:14.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T14:26:15.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:15.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:15.106+0000 7f3b2b624640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T14:26:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:15.106+0000 7f3b2b624640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T14:26:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:15.106+0000 7f3b2b624640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T14:26:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:15.106+0000 7f3b2b624640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T14:26:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:15.106+0000 7f3b2b624640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T14:26:15.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:15.107+0000 7f3b2b624640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T14:26:17.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T14:26:17.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:18.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:18.068+0000 7f7d0aec2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T14:26:18.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:18.068+0000 7f7d0aec2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T14:26:18.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:18.068+0000 7f7d0aec2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T14:26:18.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:18.068+0000 7f7d0aec2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T14:26:18.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:18.068+0000 7f7d0aec2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T14:26:18.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:18.069+0000 7f7d0aec2640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T14:26:20.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T14:26:20.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:21.032+0000 7f2d8700d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T14:26:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:21.032+0000 7f2d8700d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T14:26:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:21.032+0000 7f2d8700d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T14:26:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:21.032+0000 7f2d8700d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T14:26:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:21.032+0000 7f2d8700d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T14:26:21.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:21.032+0000 7f2d8700d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T14:26:23.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T14:26:23.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T14:26:23.944 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:23.944 INFO:tasks.ceph:role: osd.4 2024-03-25T14:26:23.944 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:26:23.944 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T14:26:24.022 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:24.022 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:24.022 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:24.022 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:24.023 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:24.023 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:24.023 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:24.023 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:24.023 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:24.023 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:24.492 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-25T14:26:24.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-25T14:26:24.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T14:26:24.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T14:26:24.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T14:26:24.741 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:24.741 INFO:tasks.ceph:role: osd.5 2024-03-25T14:26:24.741 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:26:24.741 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:24.818 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:25.291 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-25T14:26:25.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-25T14:26:25.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T14:26:25.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T14:26:25.421 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T14:26:25.489 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:25.489 INFO:tasks.ceph:role: osd.6 2024-03-25T14:26:25.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:26:25.490 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:25.565 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:25.566 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:25.566 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:25.566 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:26.059 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-25T14:26:26.061 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-25T14:26:26.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T14:26:26.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T14:26:26.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T14:26:26.278 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:26:26.278 INFO:tasks.ceph:role: osd.7 2024-03-25T14:26:26.278 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:26:26.279 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T14:26:26.355 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:26:26.355 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:26:26.355 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:26:26.355 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:26:26.356 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:26:26.356 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:26:26.356 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:26:26.356 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:26:26.356 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:26:26.356 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:26:26.880 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-25T14:26:26.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-25T14:26:26.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T14:26:27.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T14:26:27.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:27.151+0000 7fb18b0b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T14:26:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:27.151+0000 7fb18b0b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T14:26:27.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:27.151+0000 7fb18b0b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T14:26:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:27.151+0000 7fb18b0b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T14:26:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:27.151+0000 7fb18b0b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T14:26:27.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:27.152+0000 7fb18b0b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T14:26:29.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T14:26:30.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:30.114+0000 7fe0e7010640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T14:26:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:30.115+0000 7fe0e7010640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T14:26:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:30.115+0000 7fe0e7010640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T14:26:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:30.115+0000 7fe0e7010640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T14:26:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:30.115+0000 7fe0e7010640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T14:26:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:30.115+0000 7fe0e7010640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T14:26:32.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T14:26:32.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:33.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:33.077+0000 7f90f2d81640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T14:26:33.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:33.077+0000 7f90f2d81640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T14:26:33.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:33.077+0000 7f90f2d81640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T14:26:33.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:33.077+0000 7f90f2d81640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T14:26:33.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:33.077+0000 7f90f2d81640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T14:26:33.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:33.077+0000 7f90f2d81640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T14:26:35.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T14:26:35.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:36.046+0000 7f71d1a68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:36.047+0000 7f71d1a68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:36.047+0000 7f71d1a68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:36.047+0000 7f71d1a68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:36.047+0000 7f71d1a68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T14:26:36.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:26:36.047+0000 7f71d1a68640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T14:26:38.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T14:26:38.931 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T14:26:38.931 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:38.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T14:26:38.963 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:38.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T14:26:39.032 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:39.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T14:26:39.103 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:39.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T14:26:39.171 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:39.171 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T14:26:39.242 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:39.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T14:26:39.274 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:39.274 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T14:26:39.344 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:39.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T14:26:39.413 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:39.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T14:26:39.482 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:39.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T14:26:39.553 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:26:39.553 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T14:26:39.568 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:26:39.568 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T14:26:39.608 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T14:26:39.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T14:26:39.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.y] 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCCiQFm6Nq9JhAAs/iNKYtWwLf/RXVM5ZFO6Q== 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCEiQFmHR6rCBAAsbET3PeJyd3ORsMydtQLAQ== 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCHiQFm2ZZ1BhAAUqCMbLif49AfyOo9tzlhew== 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCKiQFmPCwlBBAAxTiv1QUwXbqW5QTagFnMIQ== 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCNiQFmjHX7ARAAlUDVUeOUXgNgD+TZ7oaZVQ== 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-03-25T14:26:39.651 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCCiQFmpBjWKhAAwXoQmqi7OIFJzBSIDpbfKw== 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCTiQFmoMcSCRAA/p2YDq5t6dtTDc6/mvbhqw== 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCWiQFmqeLmBhAAaL85fpnKEUJiqwz/9f9BMw== 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.6] 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCZiQFmFU6mBBAAyzcdzDbAcNzNXyGn5Gc2Ug== 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.7] 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCciQFmrtvWAhAA3voyg1nmRXIiNDOzfAC1cA== 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCCiQFmfYt/LRAA559Jmb4++/nKoi++qpQjdA== 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout:[client.1] 2024-03-25T14:26:39.652 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCCiQFmm1zhMBAAntz90YWxRcB6KIwAAglJzQ== 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCCiQFm6Nq9JhAAs/iNKYtWwLf/RXVM5ZFO6Q== 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCEiQFmHR6rCBAAsbET3PeJyd3ORsMydtQLAQ== 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCHiQFm2ZZ1BhAAUqCMbLif49AfyOo9tzlhew== 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-03-25T14:26:39.674 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCKiQFmPCwlBBAAxTiv1QUwXbqW5QTagFnMIQ== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCNiQFmjHX7ARAAlUDVUeOUXgNgD+TZ7oaZVQ== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCCiQFmpBjWKhAAwXoQmqi7OIFJzBSIDpbfKw== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCTiQFmoMcSCRAA/p2YDq5t6dtTDc6/mvbhqw== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCWiQFmqeLmBhAAaL85fpnKEUJiqwz/9f9BMw== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCZiQFmFU6mBBAAyzcdzDbAcNzNXyGn5Gc2Ug== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCciQFmrtvWAhAA3voyg1nmRXIiNDOzfAC1cA== 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-03-25T14:26:39.675 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCCiQFmfYt/LRAA559Jmb4++/nKoi++qpQjdA== 2024-03-25T14:26:39.676 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-03-25T14:26:39.676 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCCiQFmm1zhMBAAntz90YWxRcB6KIwAAglJzQ== 2024-03-25T14:26:39.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:26:39.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:26:39.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.930 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:39.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:26:39.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:26:40.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.156 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:26:40.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:26:40.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:26:40.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:26:40.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:26:40.427 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T14:26:40.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T14:26:40.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T14:26:40.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T14:26:40.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T14:26:40.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T14:26:40.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T14:26:40.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T14:26:40.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T14:26:40.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T14:26:40.922 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:40.924 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:26:40.976 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T14:26:40.976 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T14:26:40.977 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T14:26:40.977 DEBUG:teuthology.orchestra.run.smithi026:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T14:26:40.979 INFO:tasks.ceph.mon.a:Started 2024-03-25T14:26:40.979 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T14:26:40.979 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T14:26:40.980 DEBUG:teuthology.orchestra.run.smithi026:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T14:26:40.981 INFO:tasks.ceph.mon.c:Started 2024-03-25T14:26:40.982 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T14:26:40.982 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T14:26:40.982 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T14:26:41.019 INFO:tasks.ceph.mon.b:Started 2024-03-25T14:26:41.019 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T14:26:41.019 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T14:26:41.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T14:26:41.022 INFO:tasks.ceph.mgr.y:Started 2024-03-25T14:26:41.023 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T14:26:41.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T14:26:41.025 INFO:tasks.ceph.mgr.x:Started 2024-03-25T14:26:41.025 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T14:26:41.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T14:26:41.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.123+0000 7f2f58cc8640 10 monclient: get_monmap_and_config 2024-03-25T14:26:41.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.123+0000 7f2f58cc8640 10 monclient: build_initial_monmap 2024-03-25T14:26:41.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.123+0000 7f2f58cc8640 10 monclient: monmap: 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:41.124674+0000 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:41.124674+0000 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:26:41.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.124+0000 7f2f58cc8640 1 Processor -- start 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.124+0000 7f2f58cc8640 1 -- start start 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.124+0000 7f2f58cc8640 10 monclient: init 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f2f540b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f2f54142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:41.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540a9d60 con 0x7f2f540b4700 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5413ff10 con 0x7f2f54142a20 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540aa1f0 con 0x7f2f540a96b0 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f58cc8640 10 monclient(hunting): authenticate will time out at 1486.227834s 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f5413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f54142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:41.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.125+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f5413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.326+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f54142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.326+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f5413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.326+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f5413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:41.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.727+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f54142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:41.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.727+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f5413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:41.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:41.727+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f5413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:42.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:42.528+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f54142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:42.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:42.528+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f5413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:42.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:42.528+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f5413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): tick 2024-03-25T14:26:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:26:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 monclient(hunting): continuing hunt 2024-03-25T14:26:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f5413f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f5413f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f54142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2f540b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f2f54142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540b4ae0 con 0x7f2f540b4700 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540a9d60 con 0x7f2f54142a20 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5413ff10 con 0x7f2f540a96b0 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f4bfff640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f51d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:44.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.125+0000 7f2f52d9d640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:44.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.326+0000 7f2f51d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:44.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.326+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:44.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.326+0000 7f2f52d9d640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:44.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.727+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:44.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.727+0000 7f2f51d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:44.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:44.727+0000 7f2f52d9d640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:45.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:45.528+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:45.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:45.528+0000 7f2f51d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:45.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:45.528+0000 7f2f52d9d640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): tick 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 monclient(hunting): continuing hunt 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f340056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:47.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f2f54142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2f540b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540aa1f0 con 0x7f2f540b4700 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540b4ae0 con 0x7f2f54142a20 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540a9d60 con 0x7f2f540a96b0 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f4bfff640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:47.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.125+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:47.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.326+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:47.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.326+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:47.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.326+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:47.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.727+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:47.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.727+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:47.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:47.727+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:48.009 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:00:06.886 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:48.316 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:00:07.193 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:48.504 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:00:07.366 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:48.505 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:00:07.366 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:48.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:48.528+0000 7f2f52d9d640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-03-25T14:26:48.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:48.528+0000 7f2f5259c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-03-25T14:26:48.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:48.528+0000 7f2f51d9b640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.121:6789/0 2024-03-25T14:26:48.830 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:00:07.692 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:48.831 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:00:07.693 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): tick 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 monclient(hunting): continuing hunt 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f2f54142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2f540b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5413ff10 con 0x7f2f540b4700 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540aa1f0 con 0x7f2f54142a20 2024-03-25T14:26:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540b4ae0 con 0x7f2f540a96b0 2024-03-25T14:26:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.125+0000 7f2f4bfff640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:50.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.182+0000 7f2f5259c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 0x7f2f340106f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:44406/0 (socket says 172.21.15.26:44406) 2024-03-25T14:26:50.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.182+0000 7f2f5259c640 1 -- 172.21.15.26:0/4159185425 learned_addr learned my addr 172.21.15.26:0/4159185425 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:26:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.524+0000 7f2f52d9d640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T14:26:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.524+0000 7f2f52d9d640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:26:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.524+0000 7f2f52d9d640 1 --1- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 0x7f2f3400ffe0 :-1 s=OPENED pgs=3 cs=1 l=1).handle_message read tag failed 2024-03-25T14:26:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.524+0000 7f2f52d9d640 1 --1- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 0x7f2f3400ffe0 :-1 s=OPENED pgs=3 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:26:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.524+0000 7f2f5159a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-03-25T14:26:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.528+0000 7f2f51d9b640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T14:26:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.528+0000 7f2f51d9b640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T14:26:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:50.528+0000 7f2f51d9b640 1 --1- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 0x7f2f34008ea0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): tick 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 monclient(hunting): continuing hunt 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.121:6789/0 conn(0x7f2f54142a20 legacy=0x7f2f340106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f540b4700 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f2f340057e0 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:53.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2f34012090 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:53.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540a9d60 con 0x7f2f34012090 2024-03-25T14:26:53.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340106f0 con 0x7f2f340057e0 2024-03-25T14:26:53.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340108d0 con 0x7f2f540a96b0 2024-03-25T14:26:53.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:53.126+0000 7f2f4bfff640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:56.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): tick 2024-03-25T14:26:56.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:26:56.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 monclient(hunting): continuing hunt 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f540a96b0 legacy=0x7f2f3400ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.121:6789/0 conn(0x7f2f340057e0 legacy=0x7f2f3400c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f34012090 legacy=0x7f2f34014480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3400ce40 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3400d950 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f2f34016cb0 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f34010b00 con 0x7f2f3400ce40 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f34010060 con 0x7f2f34016cb0 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f34010290 con 0x7f2f3400d950 2024-03-25T14:26:56.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:56.126+0000 7f2f4bfff640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): tick 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 monclient(hunting): continuing hunt 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f3400d950 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.121:6789/0 conn(0x7f2f34016cb0 legacy=0x7f2f34014b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f3400ce40 legacy=0x7f2f3400d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f2f34015340 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:59.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3400ce40 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:59.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3400d950 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:59.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340104c0 con 0x7f2f3400ce40 2024-03-25T14:26:59.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f34015810 con 0x7f2f34015340 2024-03-25T14:26:59.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3400c720 con 0x7f2f3400d950 2024-03-25T14:26:59.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.126+0000 7f2f4bfff640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.358+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009294796 0 0) 0x7f2f340104c0 con 0x7f2f3400ce40 2024-03-25T14:26:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.358+0000 7f2f5159a640 10 monclient(hunting): _negotiate 2024-03-25T14:26:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.358+0000 7f2f5159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.358+0000 7f2f5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.358+0000 7f2f5159a640 10 monclient(hunting): my global_id is 4104 2024-03-25T14:26:59.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.358+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f40003520 con 0x7f2f3400ce40 2024-03-25T14:26:59.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.391+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700160648 0 0) 0x7f2f34015810 con 0x7f2f34015340 2024-03-25T14:26:59.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.391+0000 7f2f5159a640 10 monclient(hunting): _negotiate 2024-03-25T14:26:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.391+0000 7f2f5159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.391+0000 7f2f5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.391+0000 7f2f5159a640 10 monclient(hunting): my global_id is 4105 2024-03-25T14:26:59.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.391+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f340104c0 con 0x7f2f34015340 2024-03-25T14:26:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.393+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881730191 0 0) 0x7f2f3400c720 con 0x7f2f3400d950 2024-03-25T14:26:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.393+0000 7f2f5159a640 10 monclient(hunting): _negotiate 2024-03-25T14:26:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.393+0000 7f2f5159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.393+0000 7f2f5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.393+0000 7f2f5159a640 10 monclient(hunting): my global_id is 4106 2024-03-25T14:26:59.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.393+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f34015810 con 0x7f2f3400d950 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1037184800 0 0) 0x7f2f40003520 con 0x7f2f3400ce40 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6790/0 conn(0x7f2f3400d950 legacy=0x7f2f34008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.121:6789/0 conn(0x7f2f34015340 legacy=0x7f2f34011060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 10 monclient: _finish_hunting 0 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 1 monclient: found mon.noname-a 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3400c8b0 con 0x7f2f3400ce40 2024-03-25T14:26:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.458+0000 7f2f5159a640 10 monclient: _finish_auth 0 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f5159a640 10 monclient: _check_auth_tickets 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2f3c0040b0 con 0x7f2f3400ce40 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f58cc8640 5 monclient: authenticate success, global_id 4104 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f58cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f5159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f5159a640 10 monclient: dump: 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:26:59.464 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:26:59.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:26:59.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:26:59.465 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:26:59.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.459+0000 7f2f58cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f3c004210 con 0x7f2f3400ce40 2024-03-25T14:26:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f5159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f5159a640 1 -- 172.21.15.26:0/4159185425 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2f3c004620 con 0x7f2f3400ce40 2024-03-25T14:26:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f58cc8640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f58cc8640 4 set_mon_vals no callback set 2024-03-25T14:26:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f5159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.508+0000 7f2f5159a640 10 monclient: dump: 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:26:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:26:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:26:59.514 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 10 monclient: shutdown 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 1 -- 172.21.15.26:0/4159185425 >> v1:172.21.15.26:6789/0 conn(0x7f2f3400ce40 legacy=0x7f2f3400d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 1 -- 172.21.15.26:0/4159185425 shutdown_connections 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 1 -- 172.21.15.26:0/4159185425 >> 172.21.15.26:0/4159185425 conn(0x7f2f540668e0 msgr2=0x7f2f54066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 1 -- 172.21.15.26:0/4159185425 shutdown_connections 2024-03-25T14:26:59.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.509+0000 7f2f58cc8640 1 -- 172.21.15.26:0/4159185425 wait complete. 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 10 monclient: build_initial_monmap 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 10 monclient: monmap: 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:26:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 1 librados: starting msgr at 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 1 librados: starting objecter 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 1 Processor -- start 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 1 -- start start 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 1 librados: setting wanted keys 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 1 librados: calling monclient init 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.510+0000 7f2f58cc8640 10 monclient: init 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3400ce40 addr v1:172.21.15.26:6790/0 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3400d950 addr v1:172.21.15.121:6789/0 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f2f34015340 addr v1:172.21.15.26:6789/0 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f541cabb0 con 0x7f2f34015340 2024-03-25T14:26:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f541cae20 con 0x7f2f3400d950 2024-03-25T14:26:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f541c9850 con 0x7f2f3400ce40 2024-03-25T14:26:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.511+0000 7f2f58cc8640 10 monclient(hunting): authenticate will time out at 1504.614253s 2024-03-25T14:26:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.518+0000 7f2f52d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2f34015340 0x7f2f541c9140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56198/0 (socket says 172.21.15.26:56198) 2024-03-25T14:26:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.518+0000 7f2f52d9d640 1 -- 172.21.15.26:0/3848143336 learned_addr learned my addr 172.21.15.26:0/3848143336 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:26:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.519+0000 7f2f4affd640 10 client.?.objecter ms_handle_connect 0x7f2f34015340 2024-03-25T14:26:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.519+0000 7f2f4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.526+0000 7f2f51d9b640 1 -- 172.21.15.26:0/3848143336 >> v1:172.21.15.121:6789/0 conn(0x7f2f3400d950 legacy=0x7f2f541c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T14:26:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.526+0000 7f2f51d9b640 1 -- 172.21.15.26:0/3848143336 >> v1:172.21.15.121:6789/0 conn(0x7f2f3400d950 legacy=0x7f2f541c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:26:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.526+0000 7f2f51d9b640 1 --1- 172.21.15.26:0/3848143336 >> v1:172.21.15.121:6789/0 conn(0x7f2f3400d950 0x7f2f541c8a30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T14:26:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.526+0000 7f2f51d9b640 1 --1- 172.21.15.26:0/3848143336 >> v1:172.21.15.121:6789/0 conn(0x7f2f3400d950 0x7f2f541c8a30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T14:26:59.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.526+0000 7f2f4affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6789/0 2024-03-25T14:26:59.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.528+0000 7f2f4affd640 10 client.?.objecter ms_handle_connect 0x7f2f3400ce40 2024-03-25T14:26:59.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.528+0000 7f2f4affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.542+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482836112 0 0) 0x7f2f541cabb0 con 0x7f2f34015340 2024-03-25T14:26:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.542+0000 7f2f4affd640 10 monclient(hunting): _negotiate 2024-03-25T14:26:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.542+0000 7f2f4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.542+0000 7f2f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.542+0000 7f2f4affd640 10 monclient(hunting): my global_id is 4113 2024-03-25T14:26:59.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.542+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f3400ca40 con 0x7f2f34015340 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3735235338 0 0) 0x7f2f3400ca40 con 0x7f2f34015340 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f541cabb0 con 0x7f2f34015340 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2f440044c0 con 0x7f2f34015340 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 10 monclient(hunting): dump: 2024-03-25T14:26:59.602 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:26:59.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.598+0000 7f2f4affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:59.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.633+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919459425 0 0) 0x7f2f541c9850 con 0x7f2f3400ce40 2024-03-25T14:26:59.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.633+0000 7f2f4affd640 10 monclient(hunting): _negotiate 2024-03-25T14:26:59.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.633+0000 7f2f4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:59.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.633+0000 7f2f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:59.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.633+0000 7f2f4affd640 10 monclient(hunting): my global_id is 4112 2024-03-25T14:26:59.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.633+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f3400ca40 con 0x7f2f3400ce40 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4009371824 0 0) 0x7f2f541cabb0 con 0x7f2f34015340 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 >> v1:172.21.15.26:6790/0 conn(0x7f2f3400ce40 legacy=0x7f2f541db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 >> v1:172.21.15.121:6789/0 conn(0x7f2f3400d950 legacy=0x7f2f541c8a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 10 monclient: _finish_hunting 0 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 1 monclient: found mon.a 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:26:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f541c99e0 con 0x7f2f34015340 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 10 monclient: _finish_auth 0 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 10 monclient: _check_auth_tickets 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 5 monclient: authenticate success, global_id 4113 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 10 monclient: _renew_subs 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5413bdb0 con 0x7f2f34015340 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 10 monclient: _renew_subs 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f541dadb0 con 0x7f2f34015340 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.642+0000 7f2f58cc8640 1 librados: init done 2024-03-25T14:26:59.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.643+0000 7f2f58cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:59.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.643+0000 7f2f58cc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:59.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.643+0000 7f2f58cc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:26:59.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.643+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f541c9bd0 con 0x7f2f34015340 2024-03-25T14:26:59.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f440048d0 con 0x7f2f34015340 2024-03-25T14:26:59.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2f44004b80 con 0x7f2f34015340 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 monclient: dump: 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:26:59.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2f44005130 con 0x7f2f34015340 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1199241402 0 0) 0x7f2f44006760 con 0x7f2f34015340 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 client.4113.objecter ms_dispatch 0x7f2f54000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f50d99640 4 set_mon_vals no callback set 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2f440069f0 con 0x7f2f34015340 2024-03-25T14:26:59.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:59.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.820+0000 7f2f4affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.906+0000 7f2f58cc8640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T14:26:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.906+0000 7f2f58cc8640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T14:26:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.906+0000 7f2f58cc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:26:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:26:59.906+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2f5404f730 con 0x7f2f34015340 2024-03-25T14:27:00.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.175+0000 7f2f4affd640 1 -- 172.21.15.26:0/3848143336 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f2f440056f0 con 0x7f2f34015340 2024-03-25T14:27:00.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.175+0000 7f2f4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T14:27:00.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.175+0000 7f2f4affd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T14:27:00.180 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-03-25T14:27:00.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 10 librados: watch_flush enter 2024-03-25T14:27:00.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 10 librados: watch_flush exit 2024-03-25T14:27:00.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T14:27:00.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 10 client.4113.objecter successfully canceled tick 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 10 monclient: shutdown 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.176+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 >> v1:172.21.15.26:6789/0 conn(0x7f2f34015340 legacy=0x7f2f541c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.177+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 shutdown_connections 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.177+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 >> 172.21.15.26:0/3848143336 conn(0x7f2f540668e0 msgr2=0x7f2f540b6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.177+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 shutdown_connections 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.177+0000 7f2f58cc8640 1 -- 172.21.15.26:0/3848143336 wait complete. 2024-03-25T14:27:00.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:00.177+0000 7f2f58cc8640 1 librados: shutdown 2024-03-25T14:27:00.192 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T14:27:00.192 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T14:27:00.193 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:27:00.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T14:27:00.214 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:00.210+0000 7fea9aa38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T14:27:00.220 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:27:00.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T14:27:00.268 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:00.264+0000 7fea9aa38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T14:27:00.291 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:27:00.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T14:27:00.360 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-25T14:27:00.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T14:27:00.417 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:00.413+0000 7fded122b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T14:27:00.428 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:27:00.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T14:27:00.455 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:27:00.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T14:27:00.478 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:00.474+0000 7fded122b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T14:27:00.522 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:27:00.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T14:27:00.588 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-25T14:27:00.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T14:27:00.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new d5de21c8-0a56-4aa3-a772-897f8c8c05db 0 2024-03-25T14:27:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.791+0000 7f595945c640 10 monclient: get_monmap_and_config 2024-03-25T14:27:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.791+0000 7f595945c640 10 monclient: build_initial_monmap 2024-03-25T14:27:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.791+0000 7f595945c640 10 monclient: monmap: 2024-03-25T14:27:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:00.792886+0000 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:00.792886+0000 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.792+0000 7f595945c640 1 Processor -- start 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.792+0000 7f595945c640 1 -- start start 2024-03-25T14:27:00.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.792+0000 7f595945c640 10 monclient: init 2024-03-25T14:27:00.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient(hunting): picked mon.noname-c con 0x7f59540b4700 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient(hunting): picked mon.noname-a con 0x7f5954142a20 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient(hunting): picked mon.noname-b con 0x7f59540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59540a9d60 con 0x7f5954142a20 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595413ff10 con 0x7f59540b4700 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59540aa1f0 con 0x7f59540a96b0 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f595945c640 10 monclient(hunting): authenticate will time out at 1264.952125s 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f5952ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f59540b4700 0x7f5954142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39482/0 (socket says 172.21.15.121:39482) 2024-03-25T14:27:00.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.793+0000 7f5952ffd640 1 -- 172.21.15.121:0/1306629627 learned_addr learned my addr 172.21.15.121:0/1306629627 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:00.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781834000 0 0) 0x7f595413ff10 con 0x7f59540b4700 2024-03-25T14:27:00.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): my global_id is 4114 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f593c003520 con 0x7f59540b4700 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261587952 0 0) 0x7f59540aa1f0 con 0x7f59540a96b0 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): my global_id is 4115 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f595413ff10 con 0x7f59540a96b0 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999115614 0 0) 0x7f59540a9d60 con 0x7f5954142a20 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:00.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:00.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T14:27:00.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.796+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59540aa1f0 con 0x7f5954142a20 2024-03-25T14:27:00.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155157967 0 0) 0x7f593c003520 con 0x7f59540b4700 2024-03-25T14:27:00.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 >> v1:172.21.15.26:6790/0 conn(0x7f59540a96b0 legacy=0x7f595413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:00.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 >> v1:172.21.15.26:6789/0 conn(0x7f5954142a20 legacy=0x7f595413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:00.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 1 monclient: found mon.noname-c 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59540b4ae0 con 0x7f59540b4700 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: _finish_auth 0 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5948001690 con 0x7f59540b4700 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f595945c640 5 monclient: authenticate success, global_id 4114 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f595945c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:00.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f5951ffb640 10 monclient: dump: 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.799+0000 7f595945c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:00.811 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:00.808+0000 7fea9aa38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T14:27:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59480017f0 con 0x7f59540b4700 2024-03-25T14:27:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f5951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f5951ffb640 1 -- 172.21.15.121:0/1306629627 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5948001c00 con 0x7f59540b4700 2024-03-25T14:27:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f5951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f5951ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:00.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f5951ffb640 10 monclient: dump: 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.831 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:00.832 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:00.832 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:00.832 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:00.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 4 set_mon_vals no callback set 2024-03-25T14:27:00.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 10 monclient: shutdown 2024-03-25T14:27:00.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:00.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 1 -- 172.21.15.121:0/1306629627 >> v1:172.21.15.121:6789/0 conn(0x7f59540b4700 legacy=0x7f5954142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:00.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 1 -- 172.21.15.121:0/1306629627 shutdown_connections 2024-03-25T14:27:00.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.825+0000 7f595945c640 1 -- 172.21.15.121:0/1306629627 >> 172.21.15.121:0/1306629627 conn(0x7f59540668e0 msgr2=0x7f5954066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:00.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.826+0000 7f595945c640 1 -- 172.21.15.121:0/1306629627 shutdown_connections 2024-03-25T14:27:00.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.826+0000 7f595945c640 1 -- 172.21.15.121:0/1306629627 wait complete. 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.826+0000 7f595945c640 10 monclient: build_initial_monmap 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.826+0000 7f595945c640 10 monclient: monmap: 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:00.839 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.826+0000 7f595945c640 1 librados: starting msgr at 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.826+0000 7f595945c640 1 librados: starting objecter 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 Processor -- start 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 -- start start 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 librados: setting wanted keys 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 librados: calling monclient init 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient: init 2024-03-25T14:27:00.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient(hunting): picked mon.noname-c con 0x7f59540a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient(hunting): picked mon.noname-b con 0x7f59540b4700 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient(hunting): picked mon.noname-a con 0x7f5954142a20 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595414eca0 con 0x7f5954142a20 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59541d74d0 con 0x7f59540b4700 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59541d8350 con 0x7f59540a96b0 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.827+0000 7f595945c640 10 monclient(hunting): authenticate will time out at 1264.986795s 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f59527fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f59540b4700 0x7f59540acb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39484/0 (socket says 172.21.15.121:39484) 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f59527fc640 1 -- 172.21.15.121:0/1547125768 learned_addr learned my addr 172.21.15.121:0/1547125768 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f593b7fe640 10 client.?.objecter ms_handle_connect 0x7f59540b4700 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f593b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:00.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f593b7fe640 10 client.?.objecter ms_handle_connect 0x7f5954142a20 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f593b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f593b7fe640 10 client.?.objecter ms_handle_connect 0x7f59540a96b0 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.828+0000 7f593b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931133922 0 0) 0x7f59541d74d0 con 0x7f59540b4700 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): my global_id is 4117 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5940004eb0 con 0x7f59540b4700 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016219317 0 0) 0x7f595413ff10 con 0x7f5954142a20 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:00.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.830+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59541d74d0 con 0x7f5954142a20 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.831+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489892125 0 0) 0x7f59541d8350 con 0x7f59540a96b0 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.831+0000 7f593b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.831+0000 7f593b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.831+0000 7f593b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.831+0000 7f593b7fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.831+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f595413ff10 con 0x7f59540a96b0 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951631580 0 0) 0x7f5940004eb0 con 0x7f59540b4700 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59541d8350 con 0x7f59540b4700 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5944001a10 con 0x7f59540b4700 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:00.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): dump: 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326113140 0 0) 0x7f59541d74d0 con 0x7f5954142a20 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5940004eb0 con 0x7f5954142a20 2024-03-25T14:27:00.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f594c0019b0 con 0x7f5954142a20 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.835+0000 7f593b7fe640 10 monclient(hunting): dump: 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:00.845 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.836+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3292051009 0 0) 0x7f595413ff10 con 0x7f59540a96b0 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.836+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59541d74d0 con 0x7f59540a96b0 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.836+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5948001b00 con 0x7f59540a96b0 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.836+0000 7f593b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.836+0000 7f593b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.836+0000 7f593b7fe640 10 monclient(hunting): dump: 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.847 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2510184247 0 0) 0x7f5940004eb0 con 0x7f5954142a20 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 >> v1:172.21.15.26:6790/0 conn(0x7f59540a96b0 legacy=0x7f59541d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 >> v1:172.21.15.121:6789/0 conn(0x7f59540b4700 legacy=0x7f59540acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 1 monclient: found mon.a 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:00.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59540abd90 con 0x7f5954142a20 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f593b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 5 monclient: authenticate success, global_id 4119 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 10 monclient: _renew_subs 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59541d7720 con 0x7f5954142a20 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 10 monclient: _renew_subs 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59540ad240 con 0x7f5954142a20 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.838+0000 7f595945c640 1 librados: init done 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.839+0000 7f595945c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.839+0000 7f595945c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:00.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.839+0000 7f595945c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:00.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.839+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59541d7900 con 0x7f5954142a20 2024-03-25T14:27:00.868 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:00.864+0000 7fea9aa38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T14:27:00.913 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:00.909+0000 7fea9aa38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f594c001ee0 con 0x7f5954142a20 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f594c003b10 con 0x7f5954142a20 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 10 monclient: dump: 2024-03-25T14:27:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f594c004000 con 0x7f5954142a20 2024-03-25T14:27:00.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.941+0000 7f59517fa640 4 set_mon_vals no callback set 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 652132140 0 0) 0x7f594c005380 con 0x7f5954142a20 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 10 client.4119.objecter ms_dispatch 0x7f5954000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f594c0043a0 con 0x7f5954142a20 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:00.944+0000 7f593b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:00.967 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:00.963+0000 7fea9aa38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T14:27:01.011 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.007+0000 7fea9aa38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T14:27:01.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.016+0000 7f595945c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d5de21c8-0a56-4aa3-a772-897f8c8c05db", "id": 0}] 2024-03-25T14:27:01.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.016+0000 7f595945c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d5de21c8-0a56-4aa3-a772-897f8c8c05db", "id": 0}] 2024-03-25T14:27:01.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.016+0000 7f595945c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:01.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.016+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d5de21c8-0a56-4aa3-a772-897f8c8c05db", "id": 0} v 0) v1 -- 0x7f595404f730 con 0x7f5954142a20 2024-03-25T14:27:01.030 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.027+0000 7fded122b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T14:27:01.089 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.085+0000 7fded122b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T14:27:01.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.106+0000 7f593b7fe640 1 -- 172.21.15.121:0/1547125768 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d5de21c8-0a56-4aa3-a772-897f8c8c05db", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3145091723 0 732707515) 0x7f594c0045e0 con 0x7f5954142a20 2024-03-25T14:27:01.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.106+0000 7f593b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d5de21c8-0a56-4aa3-a772-897f8c8c05db", "id": 0}] 2024-03-25T14:27:01.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.106+0000 7f593b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:01.111 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-03-25T14:27:01.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 10 librados: watch_flush enter 2024-03-25T14:27:01.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 10 librados: watch_flush exit 2024-03-25T14:27:01.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T14:27:01.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 10 client.4119.objecter successfully canceled tick 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 10 monclient: shutdown 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 >> v1:172.21.15.26:6789/0 conn(0x7f5954142a20 legacy=0x7f59541d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 shutdown_connections 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 >> 172.21.15.121:0/1547125768 conn(0x7f59540668e0 msgr2=0x7f59540b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.107+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 shutdown_connections 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.108+0000 7f595945c640 1 -- 172.21.15.121:0/1547125768 wait complete. 2024-03-25T14:27:01.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.108+0000 7f595945c640 1 librados: shutdown 2024-03-25T14:27:01.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 5620bbc5-68e9-4f64-b0ae-62568316e93e 1 2024-03-25T14:27:01.127 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.124+0000 7fea9aa38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T14:27:01.140 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.136+0000 7fded122b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T14:27:01.197 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.193+0000 7fded122b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T14:27:01.201 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.197+0000 7fea9aa38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T14:27:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.217+0000 7efc67e81640 10 monclient: get_monmap_and_config 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.217+0000 7efc67e81640 10 monclient: build_initial_monmap 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.217+0000 7efc67e81640 10 monclient: monmap: 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:01.218579+0000 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:01.218579+0000 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.218+0000 7efc67e81640 1 Processor -- start 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.218+0000 7efc67e81640 1 -- start start 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.218+0000 7efc67e81640 10 monclient: init 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.218+0000 7efc67e81640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.218+0000 7efc67e81640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 10 monclient(hunting): picked mon.noname-b con 0x7efc600b4700 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 10 monclient(hunting): picked mon.noname-c con 0x7efc60142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 10 monclient(hunting): picked mon.noname-a con 0x7efc600a96b0 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600a9d60 con 0x7efc600a96b0 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6013ff10 con 0x7efc60142a20 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600aa1f0 con 0x7efc600b4700 2024-03-25T14:27:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc67e81640 10 monclient(hunting): authenticate will time out at 1265.377973s 2024-03-25T14:27:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc6641e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efc600a96b0 0x7efc6013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33450/0 (socket says 172.21.15.121:33450) 2024-03-25T14:27:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.219+0000 7efc6641e640 1 -- 172.21.15.121:0/1444000466 learned_addr learned my addr 172.21.15.121:0/1444000466 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330142589 0 0) 0x7efc6013ff10 con 0x7efc60142a20 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): my global_id is 4120 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4c003520 con 0x7efc60142a20 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818501573 0 0) 0x7efc600aa1f0 con 0x7efc600b4700 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): my global_id is 4121 2024-03-25T14:27:01.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6013ff10 con 0x7efc600b4700 2024-03-25T14:27:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485420238 0 0) 0x7efc600a9d60 con 0x7efc600a96b0 2024-03-25T14:27:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 10 monclient(hunting): my global_id is 4122 2024-03-25T14:27:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.222+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc600aa1f0 con 0x7efc600a96b0 2024-03-25T14:27:01.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1282153605 0 0) 0x7efc6013ff10 con 0x7efc600b4700 2024-03-25T14:27:01.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 >> v1:172.21.15.121:6789/0 conn(0x7efc60142a20 legacy=0x7efc6013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 >> v1:172.21.15.26:6789/0 conn(0x7efc600a96b0 legacy=0x7efc6013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: _finish_hunting 0 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 1 monclient: found mon.noname-b 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b4ae0 con 0x7efc600b4700 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: _finish_auth 0 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: _check_auth_tickets 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc5c001670 con 0x7efc600b4700 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc64c1b640 10 monclient: dump: 2024-03-25T14:27:01.230 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc67e81640 5 monclient: authenticate success, global_id 4121 2024-03-25T14:27:01.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.225+0000 7efc67e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:01.243 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.239+0000 7fded122b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T14:27:01.250 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.246+0000 7fea9aa38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc5c0017d0 con 0x7efc600b4700 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc64c1b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc64c1b640 1 -- 172.21.15.121:0/1444000466 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc5c001c20 con 0x7efc600b4700 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc64c1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc64c1b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc64c1b640 10 monclient: dump: 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.256 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:01.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 4 set_mon_vals no callback set 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 10 monclient: shutdown 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 1 -- 172.21.15.121:0/1444000466 >> v1:172.21.15.26:6790/0 conn(0x7efc600b4700 legacy=0x7efc60142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 1 -- 172.21.15.121:0/1444000466 shutdown_connections 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.252+0000 7efc67e81640 1 -- 172.21.15.121:0/1444000466 >> 172.21.15.121:0/1444000466 conn(0x7efc600668e0 msgr2=0x7efc60066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.253+0000 7efc67e81640 1 -- 172.21.15.121:0/1444000466 shutdown_connections 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.253+0000 7efc67e81640 1 -- 172.21.15.121:0/1444000466 wait complete. 2024-03-25T14:27:01.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.253+0000 7efc67e81640 10 monclient: build_initial_monmap 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.253+0000 7efc67e81640 10 monclient: monmap: 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.253+0000 7efc67e81640 1 librados: starting msgr at 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.253+0000 7efc67e81640 1 librados: starting objecter 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 Processor -- start 2024-03-25T14:27:01.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 -- start start 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 librados: setting wanted keys 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 librados: calling monclient init 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient: init 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient(hunting): picked mon.noname-b con 0x7efc600a96b0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient(hunting): picked mon.noname-a con 0x7efc600b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient(hunting): picked mon.noname-c con 0x7efc60142a20 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc60131a00 con 0x7efc600b4700 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600af4b0 con 0x7efc600a96b0 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600af790 con 0x7efc60142a20 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.254+0000 7efc67e81640 10 monclient(hunting): authenticate will time out at 1265.413519s 2024-03-25T14:27:01.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc6541c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efc600b4700 0x7efc60130be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33462/0 (socket says 172.21.15.121:33462) 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc6541c640 1 -- 172.21.15.121:0/4071847118 learned_addr learned my addr 172.21.15.121:0/4071847118 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc567fc640 10 client.?.objecter ms_handle_connect 0x7efc600a96b0 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc567fc640 10 client.?.objecter ms_handle_connect 0x7efc600b4700 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc567fc640 10 client.?.objecter ms_handle_connect 0x7efc60142a20 2024-03-25T14:27:01.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.255+0000 7efc567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723964704 0 0) 0x7efc600af4b0 con 0x7efc600a96b0 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc44004eb0 con 0x7efc600a96b0 2024-03-25T14:27:01.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114388277 0 0) 0x7efc600af790 con 0x7efc60142a20 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): my global_id is 4124 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc600af4b0 con 0x7efc60142a20 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575573271 0 0) 0x7efc60131a00 con 0x7efc600b4700 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T14:27:01.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.257+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc600af790 con 0x7efc600b4700 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1686516883 0 0) 0x7efc600af4b0 con 0x7efc60142a20 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc60131a00 con 0x7efc60142a20 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc58001b50 con 0x7efc60142a20 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 10 monclient(hunting): dump: 2024-03-25T14:27:01.266 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.261+0000 7efc567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:01.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112849420 0 0) 0x7efc44004eb0 con 0x7efc600a96b0 2024-03-25T14:27:01.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc600af4b0 con 0x7efc600a96b0 2024-03-25T14:27:01.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc5c001630 con 0x7efc600a96b0 2024-03-25T14:27:01.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 10 monclient(hunting): dump: 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335916529 0 0) 0x7efc600af790 con 0x7efc600b4700 2024-03-25T14:27:01.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc44004eb0 con 0x7efc600b4700 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc50001410 con 0x7efc600b4700 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.263+0000 7efc567fc640 10 monclient(hunting): dump: 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.270 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 492025089 0 0) 0x7efc60131a00 con 0x7efc60142a20 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 >> v1:172.21.15.121:6789/0 conn(0x7efc600a96b0 legacy=0x7efc600afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 >> v1:172.21.15.26:6789/0 conn(0x7efc600b4700 legacy=0x7efc60130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 1 monclient: found mon.c 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600adaf0 con 0x7efc60142a20 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 5 monclient: authenticate success, global_id 4124 2024-03-25T14:27:01.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 10 monclient: _renew_subs 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc600b02c0 con 0x7efc60142a20 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 10 monclient: _renew_subs 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc600ae590 con 0x7efc60142a20 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.266+0000 7efc67e81640 1 librados: init done 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.267+0000 7efc67e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.267+0000 7efc67e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.267+0000 7efc67e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:01.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.267+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc600adce0 con 0x7efc60142a20 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc58001f60 con 0x7efc60142a20 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc58004d00 con 0x7efc60142a20 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 10 monclient: dump: 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc57fff640 4 set_mon_vals no callback set 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efc580052b0 con 0x7efc60142a20 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 238252973 0 0) 0x7efc580067b0 con 0x7efc60142a20 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 10 client.4124.objecter ms_dispatch 0x7efc60000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 3 client.4124.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 3 client.4124.objecter handle_osd_map decoding full epoch 3 2024-03-25T14:27:01.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.291+0000 7efc567fc640 20 client.4124.objecter dump_active .. 0 homeless 2024-03-25T14:27:01.305 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.301+0000 7fea9aa38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T14:27:01.351 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.347+0000 7fea9aa38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T14:27:01.368 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.364+0000 7fded122b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T14:27:01.442 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.438+0000 7fded122b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T14:27:01.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.471+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efc58006b90 con 0x7efc60142a20 2024-03-25T14:27:01.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.471+0000 7efc567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:01.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.471+0000 7efc567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:01.492 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.488+0000 7fded122b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T14:27:01.546 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.542+0000 7fded122b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T14:27:01.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.544+0000 7efc67e81640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5620bbc5-68e9-4f64-b0ae-62568316e93e", "id": 1}] 2024-03-25T14:27:01.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.544+0000 7efc67e81640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5620bbc5-68e9-4f64-b0ae-62568316e93e", "id": 1}] 2024-03-25T14:27:01.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.544+0000 7efc67e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:01.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.544+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd new", "uuid": "5620bbc5-68e9-4f64-b0ae-62568316e93e", "id": 1} v 0) v1 -- 0x7efc6004f730 con 0x7efc60142a20 2024-03-25T14:27:01.611 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:01.608+0000 7fded122b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T14:27:01.735 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.731+0000 7fea9aa38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T14:27:01.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.754+0000 7efc567fc640 1 -- 172.21.15.121:0/4071847118 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5620bbc5-68e9-4f64-b0ae-62568316e93e", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3192190473 0 940483276) 0x7efc58005bf0 con 0x7efc60142a20 2024-03-25T14:27:01.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.755+0000 7efc567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5620bbc5-68e9-4f64-b0ae-62568316e93e", "id": 1}] 2024-03-25T14:27:01.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.755+0000 7efc567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:01.759 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.755+0000 7efc67e81640 10 librados: watch_flush enter 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.755+0000 7efc67e81640 10 librados: watch_flush exit 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.755+0000 7efc67e81640 20 client.4124.objecter shutdown clearing up homeless session... 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.755+0000 7efc67e81640 10 client.4124.objecter successfully canceled tick 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 10 monclient: shutdown 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:01.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 >> v1:172.21.15.26:6790/0 conn(0x7efc60142a20 legacy=0x7efc601312f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 shutdown_connections 2024-03-25T14:27:01.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 >> 172.21.15.121:0/4071847118 conn(0x7efc600668e0 msgr2=0x7efc600b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:01.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 shutdown_connections 2024-03-25T14:27:01.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 1 -- 172.21.15.121:0/4071847118 wait complete. 2024-03-25T14:27:01.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.756+0000 7efc67e81640 1 librados: shutdown 2024-03-25T14:27:01.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 5536078d-b3de-4699-8956-d3f8248879ac 2 2024-03-25T14:27:01.873 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.869+0000 7fea9aa38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T14:27:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.905+0000 7f1f33c82640 10 monclient: get_monmap_and_config 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.905+0000 7f1f33c82640 10 monclient: build_initial_monmap 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.905+0000 7f1f33c82640 10 monclient: monmap: 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:01.906570+0000 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:01.906570+0000 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 1 Processor -- start 2024-03-25T14:27:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 1 -- start start 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 10 monclient: init 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 10 monclient(hunting): picked mon.noname-a con 0x7f1f2c0b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2c142a20 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2c0a96b0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0a9d60 con 0x7f1f2c0b4700 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c13ff10 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.906+0000 7f1f33c82640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0aa1f0 con 0x7f1f2c142a20 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.907+0000 7f1f33c82640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.907+0000 7f1f33c82640 10 monclient(hunting): authenticate will time out at 1266.065888s 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.907+0000 7f1f31a1e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1f2c0b4700 0x7f1f2c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33466/0 (socket says 172.21.15.121:33466) 2024-03-25T14:27:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.907+0000 7f1f31a1e640 1 -- 172.21.15.121:0/230055683 learned_addr learned my addr 172.21.15.121:0/230055683 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.909+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926796996 0 0) 0x7f1f2c13ff10 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.909+0000 7f1f30a1c640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.909+0000 7f1f30a1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.909+0000 7f1f30a1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.909+0000 7f1f30a1c640 10 monclient(hunting): my global_id is 4126 2024-03-25T14:27:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.909+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18003520 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938061879 0 0) 0x7f1f2c0aa1f0 con 0x7f1f2c142a20 2024-03-25T14:27:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): my global_id is 4127 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c13ff10 con 0x7f1f2c142a20 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508889948 0 0) 0x7f1f2c0a9d60 con 0x7f1f2c0b4700 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 10 monclient(hunting): my global_id is 4128 2024-03-25T14:27:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.910+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c0aa1f0 con 0x7f1f2c0b4700 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.912+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503116257 0 0) 0x7f1f18003520 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.912+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 >> v1:172.21.15.26:6790/0 conn(0x7f1f2c142a20 legacy=0x7f1f2c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.912+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 >> v1:172.21.15.26:6789/0 conn(0x7f1f2c0b4700 legacy=0x7f1f2c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: _finish_hunting 0 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 1 monclient: found mon.noname-c 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0b4ae0 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: _finish_auth 0 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: _check_auth_tickets 2024-03-25T14:27:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1f24001a10 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f33c82640 5 monclient: authenticate success, global_id 4126 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f33c82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f30a1c640 10 monclient: dump: 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.919 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.913+0000 7f1f33c82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:01.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f24001b70 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f30a1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:01.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f30a1c640 1 -- 172.21.15.121:0/230055683 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1f24001f80 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 4 set_mon_vals no callback set 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f30a1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f30a1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f30a1c640 10 monclient: dump: 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.921 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 10 monclient: shutdown 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 1 -- 172.21.15.121:0/230055683 >> v1:172.21.15.121:6789/0 conn(0x7f1f2c0a96b0 legacy=0x7f1f2c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 1 -- 172.21.15.121:0/230055683 shutdown_connections 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.915+0000 7f1f33c82640 1 -- 172.21.15.121:0/230055683 >> 172.21.15.121:0/230055683 conn(0x7f1f2c0668e0 msgr2=0x7f1f2c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.916+0000 7f1f33c82640 1 -- 172.21.15.121:0/230055683 shutdown_connections 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.916+0000 7f1f33c82640 1 -- 172.21.15.121:0/230055683 wait complete. 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.916+0000 7f1f33c82640 10 monclient: build_initial_monmap 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.916+0000 7f1f33c82640 10 monclient: monmap: 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.916+0000 7f1f33c82640 1 librados: starting msgr at 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.916+0000 7f1f33c82640 1 librados: starting objecter 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 Processor -- start 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 -- start start 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 librados: setting wanted keys 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 librados: calling monclient init 2024-03-25T14:27:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient: init 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2c0a96b0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2c0b4700 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient(hunting): picked mon.noname-a con 0x7f1f2c142a20 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0ad3c0 con 0x7f1f2c142a20 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0acbd0 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c0ae160 con 0x7f1f2c0b4700 2024-03-25T14:27:01.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.917+0000 7f1f33c82640 10 monclient(hunting): authenticate will time out at 1266.076488s 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f3221f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1f2c142a20 0x7f1f2c0b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33480/0 (socket says 172.21.15.121:33480) 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f3221f640 1 -- 172.21.15.121:0/3694548277 learned_addr learned my addr 172.21.15.121:0/3694548277 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f227fc640 10 client.?.objecter ms_handle_connect 0x7f1f2c0a96b0 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f227fc640 10 client.?.objecter ms_handle_connect 0x7f1f2c142a20 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f227fc640 10 client.?.objecter ms_handle_connect 0x7f1f2c0b4700 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.918+0000 7f1f227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654510153 0 0) 0x7f1f2c0ae160 con 0x7f1f2c0b4700 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): my global_id is 4130 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f10004eb0 con 0x7f1f2c0b4700 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962152109 0 0) 0x7f1f2c0acbd0 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 10 monclient(hunting): my global_id is 4129 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c0ae160 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.920+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252466153 0 0) 0x7f1f2c0ad3c0 con 0x7f1f2c142a20 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.921+0000 7f1f227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.921+0000 7f1f227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.921+0000 7f1f227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.921+0000 7f1f227fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T14:27:01.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.921+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c0acbd0 con 0x7f1f2c142a20 2024-03-25T14:27:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249912147 0 0) 0x7f1f10004eb0 con 0x7f1f2c0b4700 2024-03-25T14:27:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f2c0ad3c0 con 0x7f1f2c0b4700 2024-03-25T14:27:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1f1c001910 con 0x7f1f2c0b4700 2024-03-25T14:27:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:01.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): dump: 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 811121974 0 0) 0x7f1f2c0ae160 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f10004eb0 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1f28001870 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.925+0000 7f1f227fc640 10 monclient(hunting): dump: 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.926+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 883959439 0 0) 0x7f1f2c0acbd0 con 0x7f1f2c142a20 2024-03-25T14:27:01.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.926+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f2c0ae160 con 0x7f1f2c142a20 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.926+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1f24001e80 con 0x7f1f2c142a20 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.926+0000 7f1f227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.926+0000 7f1f227fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.926+0000 7f1f227fc640 10 monclient(hunting): dump: 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:01.933 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 258279333 0 0) 0x7f1f10004eb0 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 >> v1:172.21.15.26:6790/0 conn(0x7f1f2c0b4700 legacy=0x7f1f2c139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 >> v1:172.21.15.26:6789/0 conn(0x7f1f2c142a20 legacy=0x7f1f2c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 1 monclient: found mon.b 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c13a300 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 5 monclient: authenticate success, global_id 4129 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 10 monclient: _renew_subs 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c13a550 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 10 monclient: _renew_subs 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c0b14a0 con 0x7f1f2c0a96b0 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.928+0000 7f1f33c82640 1 librados: init done 2024-03-25T14:27:01.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.929+0000 7f1f33c82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:01.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.929+0000 7f1f33c82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:01.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.929+0000 7f1f33c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:01.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:01.929+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c0b3df0 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.003 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:01.999+0000 7fea9aa38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T14:27:02.026 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.022+0000 7fded122b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T14:27:02.049 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:02.045+0000 7fea9aa38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T14:27:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f28001d40 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1f28004a00 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 monclient: dump: 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.129 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1f28004fb0 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2850687851 0 0) 0x7f1f28006590 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 client.4129.objecter ms_dispatch 0x7f1f2c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f23fff640 4 set_mon_vals no callback set 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1f280068b0 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:02.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.124+0000 7f1f227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:02.165 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:02.162+0000 7fea9aa38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:27:02.166 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.162+0000 7fded122b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T14:27:02.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.196+0000 7f1f33c82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5536078d-b3de-4699-8956-d3f8248879ac", "id": 2}] 2024-03-25T14:27:02.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.196+0000 7f1f33c82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5536078d-b3de-4699-8956-d3f8248879ac", "id": 2}] 2024-03-25T14:27:02.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.196+0000 7f1f33c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:02.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.196+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5536078d-b3de-4699-8956-d3f8248879ac", "id": 2} v 0) v1 -- 0x7f1f2c04f730 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.265 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:02.261+0000 7fea9aa38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T14:27:02.301 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.297+0000 7fded122b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T14:27:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.328+0000 7f1f227fc640 1 -- 172.21.15.121:0/3694548277 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5536078d-b3de-4699-8956-d3f8248879ac", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 347696157 0 216599125) 0x7f1f28006240 con 0x7f1f2c0a96b0 2024-03-25T14:27:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.328+0000 7f1f227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5536078d-b3de-4699-8956-d3f8248879ac", "id": 2}] 2024-03-25T14:27:02.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.328+0000 7f1f227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:02.332 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 10 librados: watch_flush enter 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 10 librados: watch_flush exit 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 10 client.4129.objecter successfully canceled tick 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 10 monclient: shutdown 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:02.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 >> v1:172.21.15.121:6789/0 conn(0x7f1f2c0a96b0 legacy=0x7f1f2c0ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 shutdown_connections 2024-03-25T14:27:02.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 >> 172.21.15.121:0/3694548277 conn(0x7f1f2c0668e0 msgr2=0x7f1f2c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:02.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 shutdown_connections 2024-03-25T14:27:02.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 1 -- 172.21.15.121:0/3694548277 wait complete. 2024-03-25T14:27:02.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.329+0000 7f1f33c82640 1 librados: shutdown 2024-03-25T14:27:02.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 558cf5ab-35b6-42a3-9e68-1d17ee5dae34 3 2024-03-25T14:27:02.352 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.348+0000 7fded122b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T14:27:02.430 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:02.426+0000 7fea9aa38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:27:02.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.439+0000 7f36fe15d640 10 monclient: get_monmap_and_config 2024-03-25T14:27:02.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.439+0000 7f36fe15d640 10 monclient: build_initial_monmap 2024-03-25T14:27:02.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.439+0000 7f36fe15d640 10 monclient: monmap: 2024-03-25T14:27:02.443 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:02.443 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.443 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:02.440350+0000 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:02.440350+0000 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:02.444 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 1 Processor -- start 2024-03-25T14:27:02.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 1 -- start start 2024-03-25T14:27:02.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient: init 2024-03-25T14:27:02.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient(hunting): picked mon.noname-a con 0x7f36f80b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient(hunting): picked mon.noname-c con 0x7f36f8142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient(hunting): picked mon.noname-b con 0x7f36f80a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80a9d60 con 0x7f36f80b4700 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f813ff10 con 0x7f36f8142a20 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80aa1f0 con 0x7f36f80a96b0 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.440+0000 7f36fe15d640 10 monclient(hunting): authenticate will time out at 1266.599621s 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.441+0000 7f36f77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f36f80b4700 0x7f36f8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33490/0 (socket says 172.21.15.121:33490) 2024-03-25T14:27:02.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.441+0000 7f36f77fe640 1 -- 172.21.15.121:0/2558774818 learned_addr learned my addr 172.21.15.121:0/2558774818 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242813832 0 0) 0x7f36f813ff10 con 0x7f36f8142a20 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): my global_id is 4132 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e0003520 con 0x7f36f8142a20 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894207259 0 0) 0x7f36f80aa1f0 con 0x7f36f80a96b0 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T14:27:02.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f813ff10 con 0x7f36f80a96b0 2024-03-25T14:27:02.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622007712 0 0) 0x7f36f80a9d60 con 0x7f36f80b4700 2024-03-25T14:27:02.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 10 monclient(hunting): my global_id is 4134 2024-03-25T14:27:02.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.444+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f80aa1f0 con 0x7f36f80b4700 2024-03-25T14:27:02.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3207071199 0 0) 0x7f36f813ff10 con 0x7f36f80a96b0 2024-03-25T14:27:02.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 >> v1:172.21.15.121:6789/0 conn(0x7f36f8142a20 legacy=0x7f36f813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 >> v1:172.21.15.26:6789/0 conn(0x7f36f80b4700 legacy=0x7f36f8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 1 monclient: found mon.noname-b 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80b4ae0 con 0x7f36f80a96b0 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f36f00019f0 con 0x7f36f80a96b0 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36f67fc640 10 monclient: dump: 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36fe15d640 5 monclient: authenticate success, global_id 4133 2024-03-25T14:27:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.447+0000 7f36fe15d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f0001b50 con 0x7f36f80a96b0 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36f67fc640 1 -- 172.21.15.121:0/2558774818 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f36f0001f60 con 0x7f36f80a96b0 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36fe15d640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36fe15d640 4 set_mon_vals no callback set 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.449+0000 7f36f67fc640 10 monclient: dump: 2024-03-25T14:27:02.457 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 10 monclient: shutdown 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 1 -- 172.21.15.121:0/2558774818 >> v1:172.21.15.26:6790/0 conn(0x7f36f80a96b0 legacy=0x7f36f813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 1 -- 172.21.15.121:0/2558774818 shutdown_connections 2024-03-25T14:27:02.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 1 -- 172.21.15.121:0/2558774818 >> 172.21.15.121:0/2558774818 conn(0x7f36f80668e0 msgr2=0x7f36f8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 1 -- 172.21.15.121:0/2558774818 shutdown_connections 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 1 -- 172.21.15.121:0/2558774818 wait complete. 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 10 monclient: build_initial_monmap 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.450+0000 7f36fe15d640 10 monclient: monmap: 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:02.459 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 librados: starting msgr at 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 librados: starting objecter 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 Processor -- start 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 -- start start 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 librados: setting wanted keys 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 librados: calling monclient init 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient: init 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient(hunting): picked mon.noname-c con 0x7f36f80a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient(hunting): picked mon.noname-a con 0x7f36f80b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient(hunting): picked mon.noname-b con 0x7f36f8142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f81349a0 con 0x7f36f80b4700 2024-03-25T14:27:02.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80aba20 con 0x7f36f8142a20 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f814e6c0 con 0x7f36f80a96b0 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.451+0000 7f36fe15d640 10 monclient(hunting): authenticate will time out at 1266.610780s 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.452+0000 7f36f6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f36f80b4700 0x7f36f8133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33492/0 (socket says 172.21.15.121:33492) 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.452+0000 7f36f6ffd640 1 -- 172.21.15.121:0/1481210426 learned_addr learned my addr 172.21.15.121:0/1481210426 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.452+0000 7f36dffff640 10 client.?.objecter ms_handle_connect 0x7f36f80b4700 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.452+0000 7f36dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.452+0000 7f36dffff640 10 client.?.objecter ms_handle_connect 0x7f36f80a96b0 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.452+0000 7f36dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.453+0000 7f36dffff640 10 client.?.objecter ms_handle_connect 0x7f36f8142a20 2024-03-25T14:27:02.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.453+0000 7f36dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:02.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.454+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332808864 0 0) 0x7f36f814e6c0 con 0x7f36f80a96b0 2024-03-25T14:27:02.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.454+0000 7f36dffff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.454+0000 7f36dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.454+0000 7f36dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.454+0000 7f36dffff640 10 monclient(hunting): my global_id is 4136 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.454+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4004eb0 con 0x7f36f80a96b0 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071597626 0 0) 0x7f36f81349a0 con 0x7f36f80b4700 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): my global_id is 4137 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f814e6c0 con 0x7f36f80b4700 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801947744 0 0) 0x7f36f80aba20 con 0x7f36f8142a20 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 10 monclient(hunting): my global_id is 4135 2024-03-25T14:27:02.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.455+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f81349a0 con 0x7f36f8142a20 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142855639 0 0) 0x7f36e4004eb0 con 0x7f36f80a96b0 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f80aba20 con 0x7f36f80a96b0 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f36e8001820 con 0x7f36f80a96b0 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 10 monclient(hunting): dump: 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.464 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.459+0000 7f36dffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.460+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837656029 0 0) 0x7f36f81349a0 con 0x7f36f8142a20 2024-03-25T14:27:02.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.460+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e4004eb0 con 0x7f36f8142a20 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.460+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f36f0001e60 con 0x7f36f8142a20 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.460+0000 7f36dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.460+0000 7f36dffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.460+0000 7f36dffff640 10 monclient(hunting): dump: 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.466 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2676382158 0 0) 0x7f36f814e6c0 con 0x7f36f80b4700 2024-03-25T14:27:02.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36f81349a0 con 0x7f36f80b4700 2024-03-25T14:27:02.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f36ec001950 con 0x7f36f80b4700 2024-03-25T14:27:02.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:02.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient(hunting): dump: 2024-03-25T14:27:02.469 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.469 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.469 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3794725448 0 0) 0x7f36f80aba20 con 0x7f36f80a96b0 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 >> v1:172.21.15.121:6789/0 conn(0x7f36f8142a20 legacy=0x7f36f8134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 >> v1:172.21.15.26:6789/0 conn(0x7f36f80b4700 legacy=0x7f36f8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:02.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 monclient: found mon.c 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f814e850 con 0x7f36f80a96b0 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient: _finish_auth 0 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 5 monclient: authenticate success, global_id 4136 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 10 monclient: _renew_subs 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f813be90 con 0x7f36f80a96b0 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 10 monclient: _renew_subs 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f813ab50 con 0x7f36f80a96b0 2024-03-25T14:27:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.462+0000 7f36fe15d640 1 librados: init done 2024-03-25T14:27:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.463+0000 7f36fe15d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.463+0000 7f36fe15d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.463+0000 7f36fe15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.463+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f804f730 con 0x7f36f80a96b0 2024-03-25T14:27:02.477 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.473+0000 7fded122b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:27:02.478 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:02.474+0000 7fea9aa38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T14:27:02.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e8001d10 con 0x7f36f80a96b0 2024-03-25T14:27:02.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f36e80047e0 con 0x7f36f80a96b0 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 10 monclient: dump: 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.573 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.574 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f36e8004d90 con 0x7f36f80a96b0 2024-03-25T14:27:02.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.568+0000 7f36f5ffb640 4 set_mon_vals no callback set 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3877221408 0 0) 0x7f36e8006440 con 0x7f36f80a96b0 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 10 client.4136.objecter ms_dispatch 0x7f36f8000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 3 client.4136.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 3 client.4136.objecter handle_osd_map decoding full epoch 5 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 20 client.4136.objecter dump_active .. 0 homeless 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f36e80058b0 con 0x7f36f80a96b0 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:02.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.569+0000 7f36dffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:02.577 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.574+0000 7fded122b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T14:27:02.635 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:02.630+0000 7fea9aa38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T14:27:02.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.641+0000 7f36fe15d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "558cf5ab-35b6-42a3-9e68-1d17ee5dae34", "id": 3}] 2024-03-25T14:27:02.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.641+0000 7f36fe15d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "558cf5ab-35b6-42a3-9e68-1d17ee5dae34", "id": 3}] 2024-03-25T14:27:02.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.641+0000 7f36fe15d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.641+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd new", "uuid": "558cf5ab-35b6-42a3-9e68-1d17ee5dae34", "id": 3} v 0) v1 -- 0x7f36f8069f30 con 0x7f36f80a96b0 2024-03-25T14:27:02.759 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.755+0000 7fded122b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:27:02.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.773+0000 7f36dffff640 1 -- 172.21.15.121:0/1481210426 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "558cf5ab-35b6-42a3-9e68-1d17ee5dae34", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2729238902 0 525046306) 0x7f36e8005fb0 con 0x7f36f80a96b0 2024-03-25T14:27:02.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.773+0000 7f36dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "558cf5ab-35b6-42a3-9e68-1d17ee5dae34", "id": 3}] 2024-03-25T14:27:02.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.773+0000 7f36dffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:02.778 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-03-25T14:27:02.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 10 librados: watch_flush enter 2024-03-25T14:27:02.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 10 librados: watch_flush exit 2024-03-25T14:27:02.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 20 client.4136.objecter shutdown clearing up homeless session... 2024-03-25T14:27:02.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 10 client.4136.objecter successfully canceled tick 2024-03-25T14:27:02.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 10 monclient: shutdown 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 >> v1:172.21.15.26:6790/0 conn(0x7f36f80a96b0 legacy=0x7f36f81d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 shutdown_connections 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 >> 172.21.15.121:0/1481210426 conn(0x7f36f80668e0 msgr2=0x7f36f813d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 shutdown_connections 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 1 -- 172.21.15.121:0/1481210426 wait complete. 2024-03-25T14:27:02.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.774+0000 7f36fe15d640 1 librados: shutdown 2024-03-25T14:27:02.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new e3c91e75-ead7-4d38-9c87-b7a3105fdff3 4 2024-03-25T14:27:02.804 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.800+0000 7fded122b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T14:27:02.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.882+0000 7f0f8f96b640 10 monclient: get_monmap_and_config 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.882+0000 7f0f8f96b640 10 monclient: build_initial_monmap 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.882+0000 7f0f8f96b640 10 monclient: monmap: 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:02.883571+0000 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:02.883571+0000 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:02.887 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 1 Processor -- start 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 1 -- start start 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient: init 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f880b4700 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f88142a20 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f880a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880a9d60 con 0x7f0f88142a20 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8813ff10 con 0x7f0f880b4700 2024-03-25T14:27:02.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880aa1f0 con 0x7f0f880a96b0 2024-03-25T14:27:02.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:02.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.883+0000 7f0f8f96b640 10 monclient(hunting): authenticate will time out at 1267.042839s 2024-03-25T14:27:02.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.884+0000 7f0f8cf06640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0f88142a20 0x7f0f8813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33498/0 (socket says 172.21.15.121:33498) 2024-03-25T14:27:02.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.884+0000 7f0f8cf06640 1 -- 172.21.15.121:0/998671125 learned_addr learned my addr 172.21.15.121:0/998671125 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890400262 0 0) 0x7f0f880aa1f0 con 0x7f0f880a96b0 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): my global_id is 4139 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f74003520 con 0x7f0f880a96b0 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436198639 0 0) 0x7f0f8813ff10 con 0x7f0f880b4700 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): my global_id is 4138 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f880aa1f0 con 0x7f0f880b4700 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562328840 0 0) 0x7f0f880a9d60 con 0x7f0f88142a20 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 10 monclient(hunting): my global_id is 4140 2024-03-25T14:27:02.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.887+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f8813ff10 con 0x7f0f88142a20 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632610981 0 0) 0x7f0f74003520 con 0x7f0f880a96b0 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 >> v1:172.21.15.121:6789/0 conn(0x7f0f880b4700 legacy=0x7f0f88142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 >> v1:172.21.15.26:6789/0 conn(0x7f0f88142a20 legacy=0x7f0f8813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 1 monclient: found mon.noname-b 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880b4ae0 con 0x7f0f880a96b0 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: _finish_auth 0 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:02.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0f80001a10 con 0x7f0f880a96b0 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f7ffff640 10 monclient: dump: 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f8f96b640 5 monclient: authenticate success, global_id 4139 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.890+0000 7f0f8f96b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f80001b70 con 0x7f0f880a96b0 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f7ffff640 1 -- 172.21.15.121:0/998671125 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0f80001f80 con 0x7f0f880a96b0 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f8f96b640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f8f96b640 4 set_mon_vals no callback set 2024-03-25T14:27:02.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.892+0000 7f0f7ffff640 10 monclient: dump: 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.897 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 10 monclient: shutdown 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 1 -- 172.21.15.121:0/998671125 >> v1:172.21.15.26:6790/0 conn(0x7f0f880a96b0 legacy=0x7f0f8813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 1 -- 172.21.15.121:0/998671125 shutdown_connections 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 1 -- 172.21.15.121:0/998671125 >> 172.21.15.121:0/998671125 conn(0x7f0f880668e0 msgr2=0x7f0f88066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 1 -- 172.21.15.121:0/998671125 shutdown_connections 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 1 -- 172.21.15.121:0/998671125 wait complete. 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 10 monclient: build_initial_monmap 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.893+0000 7f0f8f96b640 10 monclient: monmap: 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:02.900 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 librados: starting msgr at 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 librados: starting objecter 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 Processor -- start 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 -- start start 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 librados: setting wanted keys 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 librados: calling monclient init 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient: init 2024-03-25T14:27:02.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f880a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f880b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f88142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880adbe0 con 0x7f0f880b4700 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880adec0 con 0x7f0f88142a20 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880ae1a0 con 0x7f0f880a96b0 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.894+0000 7f0f8f96b640 10 monclient(hunting): authenticate will time out at 1267.053744s 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f8cf06640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0f880b4700 0x7f0f88139c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33502/0 (socket says 172.21.15.121:33502) 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f8cf06640 1 -- 172.21.15.121:0/3042760961 learned_addr learned my addr 172.21.15.121:0/3042760961 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:02.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f7dffb640 10 client.?.objecter ms_handle_connect 0x7f0f880b4700 2024-03-25T14:27:02.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:02.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f7dffb640 10 client.?.objecter ms_handle_connect 0x7f0f880a96b0 2024-03-25T14:27:02.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:02.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f7dffb640 10 client.?.objecter ms_handle_connect 0x7f0f88142a20 2024-03-25T14:27:02.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.895+0000 7f0f7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.897+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062529709 0 0) 0x7f0f880ae1a0 con 0x7f0f880a96b0 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.897+0000 7f0f7dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.897+0000 7f0f7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.897+0000 7f0f7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.897+0000 7f0f7dffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c004eb0 con 0x7f0f880a96b0 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266035983 0 0) 0x7f0f880adec0 con 0x7f0f88142a20 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): my global_id is 4141 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f880ae1a0 con 0x7f0f88142a20 2024-03-25T14:27:02.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204806066 0 0) 0x7f0f880adbe0 con 0x7f0f880b4700 2024-03-25T14:27:02.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:02.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:02.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:02.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 10 monclient(hunting): my global_id is 4143 2024-03-25T14:27:02.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.898+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f880adec0 con 0x7f0f880b4700 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3800890830 0 0) 0x7f0f6c004eb0 con 0x7f0f880a96b0 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f880adbe0 con 0x7f0f880a96b0 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317072810 0 0) 0x7f0f880ae1a0 con 0x7f0f88142a20 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f6c004eb0 con 0x7f0f88142a20 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0f80001e80 con 0x7f0f88142a20 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 10 monclient(hunting): dump: 2024-03-25T14:27:02.907 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.902+0000 7f0f7dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.903+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0f84001a10 con 0x7f0f880a96b0 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.903+0000 7f0f7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.903+0000 7f0f7dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.903+0000 7f0f7dffb640 10 monclient(hunting): dump: 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.909 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2910053294 0 0) 0x7f0f880adec0 con 0x7f0f880b4700 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f880ae1a0 con 0x7f0f880b4700 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0f78001870 con 0x7f0f880b4700 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient(hunting): dump: 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:02.911 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1777776472 0 0) 0x7f0f6c004eb0 con 0x7f0f88142a20 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 >> v1:172.21.15.26:6790/0 conn(0x7f0f880a96b0 legacy=0x7f0f880ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 >> v1:172.21.15.26:6789/0 conn(0x7f0f880b4700 legacy=0x7f0f88139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient: _finish_hunting 0 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 monclient: found mon.b 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8813b200 con 0x7f0f88142a20 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient: _finish_auth 0 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 10 monclient: _check_auth_tickets 2024-03-25T14:27:02.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 5 monclient: authenticate success, global_id 4141 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 10 monclient: _renew_subs 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f880aefb0 con 0x7f0f88142a20 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 10 monclient: _renew_subs 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f880ac270 con 0x7f0f88142a20 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.905+0000 7f0f8f96b640 1 librados: init done 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.906+0000 7f0f8f96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.906+0000 7f0f8f96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.906+0000 7f0f8f96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:02.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:02.906+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f880ac520 con 0x7f0f88142a20 2024-03-25T14:27:02.972 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:02.968+0000 7fded122b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T14:27:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f80001fe0 con 0x7f0f88142a20 2024-03-25T14:27:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:03.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0f80003dc0 con 0x7f0f88142a20 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 monclient: dump: 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0f800042b0 con 0x7f0f88142a20 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7f7fe640 4 set_mon_vals no callback set 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1430106865 0 0) 0x7f0f800057d0 con 0x7f0f88142a20 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 client.4141.objecter ms_dispatch 0x7f0f88000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0f80005bb0 con 0x7f0f88142a20 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.004+0000 7f0f7dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.075+0000 7f0f8f96b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e3c91e75-ead7-4d38-9c87-b7a3105fdff3", "id": 4}] 2024-03-25T14:27:03.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.075+0000 7f0f8f96b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e3c91e75-ead7-4d38-9c87-b7a3105fdff3", "id": 4}] 2024-03-25T14:27:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.075+0000 7f0f8f96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:03.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.075+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e3c91e75-ead7-4d38-9c87-b7a3105fdff3", "id": 4} v 0) v1 -- 0x7f0f8804f730 con 0x7f0f88142a20 2024-03-25T14:27:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.142+0000 7f0f7dffb640 1 -- 172.21.15.121:0/3042760961 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3c91e75-ead7-4d38-9c87-b7a3105fdff3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 209341088 0 1697012583) 0x7f0f800052c0 con 0x7f0f88142a20 2024-03-25T14:27:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.142+0000 7f0f7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e3c91e75-ead7-4d38-9c87-b7a3105fdff3", "id": 4}] 2024-03-25T14:27:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.142+0000 7f0f7dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:03.146 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 10 librados: watch_flush enter 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 10 librados: watch_flush exit 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 10 client.4141.objecter successfully canceled tick 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 10 monclient: shutdown 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 >> v1:172.21.15.121:6789/0 conn(0x7f0f88142a20 legacy=0x7f0f8813a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 shutdown_connections 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 >> 172.21.15.121:0/3042760961 conn(0x7f0f880668e0 msgr2=0x7f0f8813c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 shutdown_connections 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 1 -- 172.21.15.121:0/3042760961 wait complete. 2024-03-25T14:27:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.143+0000 7f0f8f96b640 1 librados: shutdown 2024-03-25T14:27:03.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 51fdbe3a-8a98-458e-b558-7304df69e600 5 2024-03-25T14:27:03.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.250+0000 7f6939e95640 10 monclient: get_monmap_and_config 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.250+0000 7f6939e95640 10 monclient: build_initial_monmap 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.250+0000 7f6939e95640 10 monclient: monmap: 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:03.251825+0000 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:03.251825+0000 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:03.256 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.251+0000 7f6939e95640 1 Processor -- start 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.251+0000 7f6939e95640 1 -- start start 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.251+0000 7f6939e95640 10 monclient: init 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient(hunting): picked mon.noname-b con 0x7f693406c2d0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient(hunting): picked mon.noname-c con 0x7f693406c6b0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient(hunting): picked mon.noname-a con 0x7f6934142d90 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69340b0840 con 0x7f6934142d90 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934143200 con 0x7f693406c6b0 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693406d810 con 0x7f693406c2d0 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6939e95640 10 monclient(hunting): authenticate will time out at 1267.411132s 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6933fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6934142d90 0x7f693414dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33512/0 (socket says 172.21.15.121:33512) 2024-03-25T14:27:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.252+0000 7f6933fff640 1 -- 172.21.15.121:0/1002378215 learned_addr learned my addr 172.21.15.121:0/1002378215 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652870568 0 0) 0x7f6928001110 con 0x7f693406c6b0 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): my global_id is 4144 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c003520 con 0x7f693406c6b0 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243053633 0 0) 0x7f693406d810 con 0x7f693406c2d0 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 10 monclient(hunting): my global_id is 4145 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.254+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6928001110 con 0x7f693406c2d0 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.256+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736572023 0 0) 0x7f69340b0840 con 0x7f6934142d90 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.256+0000 7f69327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.256+0000 7f69327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.256+0000 7f69327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.256+0000 7f69327fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T14:27:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.256+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693406d810 con 0x7f6934142d90 2024-03-25T14:27:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2109200336 0 0) 0x7f691c003520 con 0x7f693406c6b0 2024-03-25T14:27:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 >> v1:172.21.15.26:6790/0 conn(0x7f693406c2d0 legacy=0x7f69340afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 >> v1:172.21.15.26:6789/0 conn(0x7f6934142d90 legacy=0x7f693414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 1 monclient: found mon.noname-c 2024-03-25T14:27:03.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69340ae260 con 0x7f693406c6b0 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6928001680 con 0x7f693406c6b0 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f69327fc640 10 monclient: dump: 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.263 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f6939e95640 5 monclient: authenticate success, global_id 4144 2024-03-25T14:27:03.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.258+0000 7f6939e95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69280017e0 con 0x7f693406c6b0 2024-03-25T14:27:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f69327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:03.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f69327fc640 1 -- 172.21.15.121:0/1002378215 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6928001b70 con 0x7f693406c6b0 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f6939e95640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f6939e95640 4 set_mon_vals no callback set 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f69327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f69327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.259+0000 7f69327fc640 10 monclient: dump: 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.268 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 10 monclient: shutdown 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 1 -- 172.21.15.121:0/1002378215 >> v1:172.21.15.121:6789/0 conn(0x7f693406c6b0 legacy=0x7f6934142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 1 -- 172.21.15.121:0/1002378215 shutdown_connections 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 1 -- 172.21.15.121:0/1002378215 >> 172.21.15.121:0/1002378215 conn(0x7f69340b5e40 msgr2=0x7f69340b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 1 -- 172.21.15.121:0/1002378215 shutdown_connections 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.260+0000 7f6939e95640 1 -- 172.21.15.121:0/1002378215 wait complete. 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 10 monclient: build_initial_monmap 2024-03-25T14:27:03.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 10 monclient: monmap: 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 1 librados: starting msgr at 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 1 librados: starting objecter 2024-03-25T14:27:03.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 1 Processor -- start 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 1 -- start start 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 1 librados: setting wanted keys 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 1 librados: calling monclient init 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.261+0000 7f6939e95640 10 monclient: init 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient(hunting): picked mon.noname-c con 0x7f693406c2d0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient(hunting): picked mon.noname-a con 0x7f693406c6b0 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient(hunting): picked mon.noname-b con 0x7f6934142d90 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934133da0 con 0x7f693406c6b0 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69341400a0 con 0x7f6934142d90 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934135ab0 con 0x7f693406c2d0 2024-03-25T14:27:03.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.262+0000 7f6939e95640 10 monclient(hunting): authenticate will time out at 1267.421379s 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.263+0000 7f69337fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f693406c2d0 0x7f6934140df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.121:47952/0 (socket says 172.21.15.121:47952) 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.263+0000 7f69337fe640 1 -- 172.21.15.121:0/3902235566 learned_addr learned my addr 172.21.15.121:0/3902235566 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.263+0000 7f691bfff640 10 client.?.objecter ms_handle_connect 0x7f693406c2d0 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.263+0000 7f691bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.263+0000 7f691bfff640 10 client.?.objecter ms_handle_connect 0x7f6934142d90 2024-03-25T14:27:03.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.263+0000 7f691bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756193798 0 0) 0x7f69341400a0 con 0x7f6934142d90 2024-03-25T14:27:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): my global_id is 4147 2024-03-25T14:27:03.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6920004eb0 con 0x7f6934142d90 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 client.?.objecter ms_handle_connect 0x7f693406c6b0 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885066252 0 0) 0x7f6934135ab0 con 0x7f693406c2d0 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 10 monclient(hunting): my global_id is 4148 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.265+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69341400a0 con 0x7f693406c2d0 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.268+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155666122 0 0) 0x7f6934133da0 con 0x7f693406c6b0 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.268+0000 7f691bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.268+0000 7f691bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.268+0000 7f691bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.268+0000 7f691bfff640 10 monclient(hunting): my global_id is 4149 2024-03-25T14:27:03.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.268+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6934135ab0 con 0x7f693406c6b0 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3818323059 0 0) 0x7f69341400a0 con 0x7f693406c2d0 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6934133da0 con 0x7f693406c2d0 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6924001910 con 0x7f693406c2d0 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): dump: 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.276 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333607248 0 0) 0x7f6920004eb0 con 0x7f6934142d90 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69341400a0 con 0x7f6934142d90 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f692c001c70 con 0x7f6934142d90 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.270+0000 7f691bfff640 10 monclient(hunting): dump: 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.278 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.279 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.279 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2808762303 0 0) 0x7f6934133da0 con 0x7f693406c2d0 2024-03-25T14:27:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 >> v1:172.21.15.121:6789/0 conn(0x7f6934142d90 legacy=0x7f69341e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 >> v1:172.21.15.26:6789/0 conn(0x7f693406c6b0 legacy=0x7f69341353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:03.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 1 monclient: found mon.c 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934135c40 con 0x7f693406c2d0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 10 monclient: _finish_auth 0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f691bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 5 monclient: authenticate success, global_id 4148 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 10 monclient: _renew_subs 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69341441c0 con 0x7f693406c2d0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 10 monclient: _renew_subs 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6934140810 con 0x7f693406c2d0 2024-03-25T14:27:03.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.273+0000 7f6939e95640 1 librados: init done 2024-03-25T14:27:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.274+0000 7f6939e95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.274+0000 7f6939e95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.274+0000 7f6939e95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.274+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693406d470 con 0x7f693406c2d0 2024-03-25T14:27:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6924003610 con 0x7f693406c2d0 2024-03-25T14:27:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6924003a20 con 0x7f693406c2d0 2024-03-25T14:27:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 10 monclient: dump: 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6924003fd0 con 0x7f693406c2d0 2024-03-25T14:27:03.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.378+0000 7f6931ffb640 4 set_mon_vals no callback set 2024-03-25T14:27:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 286956026 0 0) 0x7f6924006700 con 0x7f693406c2d0 2024-03-25T14:27:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 10 client.4148.objecter ms_dispatch 0x7f6934000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T14:27:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 3 client.4148.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T14:27:03.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 3 client.4148.objecter handle_osd_map decoding full epoch 7 2024-03-25T14:27:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-25T14:27:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6924006a50 con 0x7f693406c2d0 2024-03-25T14:27:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.381+0000 7f691bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:03.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.454+0000 7f6939e95640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "51fdbe3a-8a98-458e-b558-7304df69e600", "id": 5}] 2024-03-25T14:27:03.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.454+0000 7f6939e95640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "51fdbe3a-8a98-458e-b558-7304df69e600", "id": 5}] 2024-03-25T14:27:03.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.454+0000 7f6939e95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.454+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd new", "uuid": "51fdbe3a-8a98-458e-b558-7304df69e600", "id": 5} v 0) v1 -- 0x7f693404f730 con 0x7f693406c2d0 2024-03-25T14:27:03.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.539+0000 7f691bfff640 1 -- 172.21.15.121:0/3902235566 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "51fdbe3a-8a98-458e-b558-7304df69e600", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1342194919 0 1988412176) 0x7f6924006180 con 0x7f693406c2d0 2024-03-25T14:27:03.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.539+0000 7f691bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "51fdbe3a-8a98-458e-b558-7304df69e600", "id": 5}] 2024-03-25T14:27:03.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.539+0000 7f691bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:03.543 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-03-25T14:27:03.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 10 librados: watch_flush enter 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 10 librados: watch_flush exit 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 10 client.4148.objecter successfully canceled tick 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 10 monclient: shutdown 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 >> v1:172.21.15.26:6790/0 conn(0x7f693406c2d0 legacy=0x7f6934140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 shutdown_connections 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 >> 172.21.15.121:0/3902235566 conn(0x7f69340b5e40 msgr2=0x7f6934074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 shutdown_connections 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 1 -- 172.21.15.121:0/3902235566 wait complete. 2024-03-25T14:27:03.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.540+0000 7f6939e95640 1 librados: shutdown 2024-03-25T14:27:03.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new f32d3668-ed36-49b2-a983-3d512dc5f96f 6 2024-03-25T14:27:03.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.650+0000 7f934e996640 10 monclient: get_monmap_and_config 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.650+0000 7f934e996640 10 monclient: build_initial_monmap 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.650+0000 7f934e996640 10 monclient: monmap: 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:03.651752+0000 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:03.651752+0000 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:03.656 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:03.656 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.651+0000 7f934e996640 1 Processor -- start 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.651+0000 7f934e996640 1 -- start start 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.651+0000 7f934e996640 10 monclient: init 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient(hunting): picked mon.noname-a con 0x7f93480b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:03.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient(hunting): picked mon.noname-b con 0x7f9348142a20 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient(hunting): picked mon.noname-c con 0x7f93480a96b0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480a9d60 con 0x7f93480b4700 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934813ff10 con 0x7f93480a96b0 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480aa1f0 con 0x7f9348142a20 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934e996640 10 monclient(hunting): authenticate will time out at 1267.811039s 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934cf33640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f93480a96b0 0x7f934813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39590/0 (socket says 172.21.15.121:39590) 2024-03-25T14:27:03.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.652+0000 7f934cf33640 1 -- 172.21.15.121:0/3190749673 learned_addr learned my addr 172.21.15.121:0/3190749673 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:03.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.654+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105142914 0 0) 0x7f934813ff10 con 0x7f93480a96b0 2024-03-25T14:27:03.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.654+0000 7f9346ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.654+0000 7f9346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.654+0000 7f9346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.654+0000 7f9346ffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T14:27:03.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.654+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330003520 con 0x7f93480a96b0 2024-03-25T14:27:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.655+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745178369 0 0) 0x7f93480aa1f0 con 0x7f9348142a20 2024-03-25T14:27:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.655+0000 7f9346ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.655+0000 7f9346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.655+0000 7f9346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.655+0000 7f9346ffd640 10 monclient(hunting): my global_id is 4151 2024-03-25T14:27:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.655+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f934813ff10 con 0x7f9348142a20 2024-03-25T14:27:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242738950 0 0) 0x7f9330003520 con 0x7f93480a96b0 2024-03-25T14:27:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 >> v1:172.21.15.26:6790/0 conn(0x7f9348142a20 legacy=0x7f934813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 >> v1:172.21.15.26:6789/0 conn(0x7f93480b4700 legacy=0x7f9348142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 1 monclient: found mon.noname-c 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93480b4ae0 con 0x7f93480a96b0 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: _finish_auth 0 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9340001a10 con 0x7f93480a96b0 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f9346ffd640 10 monclient: dump: 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f934e996640 5 monclient: authenticate success, global_id 4150 2024-03-25T14:27:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.658+0000 7f934e996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9340001b70 con 0x7f93480a96b0 2024-03-25T14:27:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f9346ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f9346ffd640 1 -- 172.21.15.121:0/3190749673 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9340001f80 con 0x7f93480a96b0 2024-03-25T14:27:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f9346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f9346ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f9346ffd640 10 monclient: dump: 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f934e996640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:03.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.667+0000 7f934e996640 4 set_mon_vals no callback set 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 10 monclient: shutdown 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 1 -- 172.21.15.121:0/3190749673 >> v1:172.21.15.121:6789/0 conn(0x7f93480a96b0 legacy=0x7f934813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 1 -- 172.21.15.121:0/3190749673 shutdown_connections 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 1 -- 172.21.15.121:0/3190749673 >> 172.21.15.121:0/3190749673 conn(0x7f93480668e0 msgr2=0x7f9348066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 1 -- 172.21.15.121:0/3190749673 shutdown_connections 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.668+0000 7f934e996640 1 -- 172.21.15.121:0/3190749673 wait complete. 2024-03-25T14:27:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 10 monclient: build_initial_monmap 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 10 monclient: monmap: 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 1 librados: starting msgr at 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 1 librados: starting objecter 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 1 Processor -- start 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 1 -- start start 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 1 librados: setting wanted keys 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 1 librados: calling monclient init 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.669+0000 7f934e996640 10 monclient: init 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:27:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient(hunting): picked mon.noname-b con 0x7f93480a96b0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient(hunting): picked mon.noname-c con 0x7f93480b4700 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient(hunting): picked mon.noname-a con 0x7f9348142a20 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934813b3e0 con 0x7f9348142a20 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480ae520 con 0x7f93480a96b0 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480ab880 con 0x7f93480b4700 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f934e996640 10 monclient(hunting): authenticate will time out at 1267.829132s 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f9347fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f93480a96b0 0x7f93480ac810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:39600/0 (socket says 172.21.15.121:39600) 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f9347fff640 1 -- 172.21.15.121:0/3481450577 learned_addr learned my addr 172.21.15.121:0/3481450577 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.670+0000 7f9344ff9640 10 client.?.objecter ms_handle_connect 0x7f93480a96b0 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.671+0000 7f9344ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.671+0000 7f9344ff9640 10 client.?.objecter ms_handle_connect 0x7f93480b4700 2024-03-25T14:27:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.671+0000 7f9344ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.679+0000 7f9344ff9640 10 client.?.objecter ms_handle_connect 0x7f9348142a20 2024-03-25T14:27:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.679+0000 7f9344ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:03.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.710+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581546459 0 0) 0x7f93480ab880 con 0x7f93480b4700 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.710+0000 7f9344ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.710+0000 7f9344ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.710+0000 7f9344ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.710+0000 7f9344ff9640 10 monclient(hunting): my global_id is 4154 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.710+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334004eb0 con 0x7f93480b4700 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.711+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617871940 0 0) 0x7f934813b3e0 con 0x7f9348142a20 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.711+0000 7f9344ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.711+0000 7f9344ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.711+0000 7f9344ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.711+0000 7f9344ff9640 10 monclient(hunting): my global_id is 4155 2024-03-25T14:27:03.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.711+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93480ab880 con 0x7f9348142a20 2024-03-25T14:27:03.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.713+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892841341 0 0) 0x7f93480ae520 con 0x7f93480a96b0 2024-03-25T14:27:03.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.713+0000 7f9344ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:03.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.713+0000 7f9344ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:03.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.713+0000 7f9344ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:03.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.713+0000 7f9344ff9640 10 monclient(hunting): my global_id is 4153 2024-03-25T14:27:03.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.713+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f934813b3e0 con 0x7f93480a96b0 2024-03-25T14:27:03.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1839062609 0 0) 0x7f9334004eb0 con 0x7f93480b4700 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93480ae520 con 0x7f93480b4700 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9338001930 con 0x7f93480b4700 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 10 monclient(hunting): dump: 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.722 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.717+0000 7f9344ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.718+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422755295 0 0) 0x7f93480ab880 con 0x7f9348142a20 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.718+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9334004eb0 con 0x7f9348142a20 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.718+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9340001e80 con 0x7f9348142a20 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.718+0000 7f9344ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.718+0000 7f9344ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.718+0000 7f9344ff9640 10 monclient(hunting): dump: 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.723 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.724 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.719+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450705823 0 0) 0x7f934813b3e0 con 0x7f93480a96b0 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.719+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93480ab880 con 0x7f93480a96b0 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.719+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f933c001970 con 0x7f93480a96b0 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.719+0000 7f9344ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.719+0000 7f9344ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.719+0000 7f9344ff9640 10 monclient(hunting): dump: 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.726 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 975773539 0 0) 0x7f93480ae520 con 0x7f93480b4700 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 >> v1:172.21.15.121:6789/0 conn(0x7f93480a96b0 legacy=0x7f93480ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 >> v1:172.21.15.26:6789/0 conn(0x7f9348142a20 legacy=0x7f93480b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 1 monclient: found mon.c 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93480aba10 con 0x7f93480b4700 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 10 monclient: _finish_auth 0 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f9344ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 5 monclient: authenticate success, global_id 4154 2024-03-25T14:27:03.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 10 monclient: _renew_subs 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934813a490 con 0x7f93480b4700 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 10 monclient: _renew_subs 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93480ad430 con 0x7f93480b4700 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.721+0000 7f934e996640 1 librados: init done 2024-03-25T14:27:03.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.723+0000 7f934e996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.723+0000 7f934e996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.723+0000 7f934e996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.723+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934804f730 con 0x7f93480b4700 2024-03-25T14:27:03.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93380035f0 con 0x7f93480b4700 2024-03-25T14:27:03.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:03.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f93467fc640 4 set_mon_vals no callback set 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9338003900 con 0x7f93480b4700 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 10 monclient: dump: 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:03.731 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9338003df0 con 0x7f93480b4700 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 889650581 0 0) 0x7f93380056d0 con 0x7f93480b4700 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 10 client.4154.objecter ms_dispatch 0x7f9348000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 3 client.4154.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 3 client.4154.objecter handle_osd_map decoding full epoch 8 2024-03-25T14:27:03.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.726+0000 7f9344ff9640 20 client.4154.objecter dump_active .. 0 homeless 2024-03-25T14:27:03.736 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:03.733+0000 7fea9aa38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T14:27:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.826+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f93380051d0 con 0x7f93480b4700 2024-03-25T14:27:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.826+0000 7f9344ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.826+0000 7f9344ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:03.833 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:03.829+0000 7fded122b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T14:27:03.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.898+0000 7f934e996640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f32d3668-ed36-49b2-a983-3d512dc5f96f", "id": 6}] 2024-03-25T14:27:03.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.898+0000 7f934e996640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f32d3668-ed36-49b2-a983-3d512dc5f96f", "id": 6}] 2024-03-25T14:27:03.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.898+0000 7f934e996640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:03.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.898+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f32d3668-ed36-49b2-a983-3d512dc5f96f", "id": 6} v 0) v1 -- 0x7f9348142220 con 0x7f93480b4700 2024-03-25T14:27:03.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.990+0000 7f9344ff9640 1 -- 172.21.15.121:0/3481450577 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f32d3668-ed36-49b2-a983-3d512dc5f96f", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3132927809 0 1113811849) 0x7f93380054f0 con 0x7f93480b4700 2024-03-25T14:27:03.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.990+0000 7f9344ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f32d3668-ed36-49b2-a983-3d512dc5f96f", "id": 6}] 2024-03-25T14:27:03.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f9344ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:03.995 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-03-25T14:27:03.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 10 librados: watch_flush enter 2024-03-25T14:27:03.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 10 librados: watch_flush exit 2024-03-25T14:27:03.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 20 client.4154.objecter shutdown clearing up homeless session... 2024-03-25T14:27:03.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 10 client.4154.objecter successfully canceled tick 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 10 monclient: shutdown 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 >> v1:172.21.15.26:6790/0 conn(0x7f93480b4700 legacy=0x7f9348139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 shutdown_connections 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.991+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 >> 172.21.15.121:0/3481450577 conn(0x7f93480668e0 msgr2=0x7f934813c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.992+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 shutdown_connections 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.992+0000 7f934e996640 1 -- 172.21.15.121:0/3481450577 wait complete. 2024-03-25T14:27:03.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:03.992+0000 7f934e996640 1 librados: shutdown 2024-03-25T14:27:04.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new ac269fe1-ef8f-4fe9-80e6-3081e6131db8 7 2024-03-25T14:27:04.018 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:04.014+0000 7fea9aa38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T14:27:04.072 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:04.068+0000 7fea9aa38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T14:27:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.104+0000 7f9ecb41e640 10 monclient: get_monmap_and_config 2024-03-25T14:27:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.104+0000 7f9ecb41e640 10 monclient: build_initial_monmap 2024-03-25T14:27:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.104+0000 7f9ecb41e640 10 monclient: monmap: 2024-03-25T14:27:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:27:04.105861+0000 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:27:04.105861+0000 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.105+0000 7f9ecb41e640 1 Processor -- start 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.105+0000 7f9ecb41e640 1 -- start start 2024-03-25T14:27:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.105+0000 7f9ecb41e640 10 monclient: init 2024-03-25T14:27:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient(hunting): picked mon.noname-c con 0x7f9ec40b4700 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ec4142a20 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient(hunting): picked mon.noname-b con 0x7f9ec40a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec40a9d60 con 0x7f9ec4142a20 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec413ff10 con 0x7f9ec40b4700 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec40aa1f0 con 0x7f9ec40a96b0 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ecb41e640 10 monclient(hunting): authenticate will time out at 1268.265189s 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ec89b9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9ec4142a20 0x7f9ec413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33552/0 (socket says 172.21.15.121:33552) 2024-03-25T14:27:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.106+0000 7f9ec89b9640 1 -- 172.21.15.121:0/2300359264 learned_addr learned my addr 172.21.15.121:0/2300359264 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.108+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326470325 0 0) 0x7f9ec40a9d60 con 0x7f9ec4142a20 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.108+0000 7f9ebbfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.108+0000 7f9ebbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.108+0000 7f9ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.108+0000 7f9ebbfff640 10 monclient(hunting): my global_id is 4158 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.108+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb0003520 con 0x7f9ec4142a20 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047957729 0 0) 0x7f9ec413ff10 con 0x7f9ec40b4700 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:04.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:04.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:04.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): my global_id is 4156 2024-03-25T14:27:04.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec40a9d60 con 0x7f9ec40b4700 2024-03-25T14:27:04.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043189848 0 0) 0x7f9ec40aa1f0 con 0x7f9ec40a96b0 2024-03-25T14:27:04.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:04.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:04.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:04.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.109+0000 7f9ebbfff640 10 monclient(hunting): my global_id is 4157 2024-03-25T14:27:04.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.110+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec413ff10 con 0x7f9ec40a96b0 2024-03-25T14:27:04.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2807154464 0 0) 0x7f9ec40a9d60 con 0x7f9ec40b4700 2024-03-25T14:27:04.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 >> v1:172.21.15.26:6790/0 conn(0x7f9ec40a96b0 legacy=0x7f9ec413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:04.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 >> v1:172.21.15.26:6789/0 conn(0x7f9ec4142a20 legacy=0x7f9ec413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:04.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:04.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 1 monclient: found mon.noname-c 2024-03-25T14:27:04.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec40b4ae0 con 0x7f9ec40b4700 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: _finish_auth 0 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9eb4001670 con 0x7f9ec40b4700 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ebbfff640 10 monclient: dump: 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:04.118 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ecb41e640 5 monclient: authenticate success, global_id 4156 2024-03-25T14:27:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.112+0000 7f9ecb41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9eb40017d0 con 0x7f9ec40b4700 2024-03-25T14:27:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ebbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ebbfff640 1 -- 172.21.15.121:0/2300359264 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9eb4001be0 con 0x7f9ec40b4700 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 4 set_mon_vals no callback set 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ebbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ebbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ebbfff640 10 monclient: dump: 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 10 monclient: shutdown 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2300359264 >> v1:172.21.15.121:6789/0 conn(0x7f9ec40b4700 legacy=0x7f9ec4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2300359264 shutdown_connections 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2300359264 >> 172.21.15.121:0/2300359264 conn(0x7f9ec40668e0 msgr2=0x7f9ec4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2300359264 shutdown_connections 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.114+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2300359264 wait complete. 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.115+0000 7f9ecb41e640 10 monclient: build_initial_monmap 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.115+0000 7f9ecb41e640 10 monclient: monmap: 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-25T14:27:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.115+0000 7f9ecb41e640 1 librados: starting msgr at 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.115+0000 7f9ecb41e640 1 librados: starting objecter 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 Processor -- start 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 -- start start 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 librados: setting wanted keys 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 librados: calling monclient init 2024-03-25T14:27:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient: init 2024-03-25T14:27:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient(hunting): picked mon.noname-c con 0x7f9ec40a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ec40b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient(hunting): picked mon.noname-b con 0x7f9ec4142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec4135f10 con 0x7f9ec40b4700 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec4139d70 con 0x7f9ec4142a20 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec413a050 con 0x7f9ec40a96b0 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.116+0000 7f9ecb41e640 10 monclient(hunting): authenticate will time out at 1268.275557s 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9ec89b9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9ec40b4700 0x7f9ec41db320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.121:33556/0 (socket says 172.21.15.121:33556) 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9ec89b9640 1 -- 172.21.15.121:0/2018994185 learned_addr learned my addr 172.21.15.121:0/2018994185 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9eb9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ec40b4700 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9eb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9eb9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ec4142a20 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9eb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9eb9ffb640 10 client.?.objecter ms_handle_connect 0x7f9ec40a96b0 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.117+0000 7f9eb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898232063 0 0) 0x7f9ec4139d70 con 0x7f9ec4142a20 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): my global_id is 4159 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea8004eb0 con 0x7f9ec4142a20 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35383826 0 0) 0x7f9ec4135f10 con 0x7f9ec40b4700 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 10 monclient(hunting): my global_id is 4161 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.119+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec4139d70 con 0x7f9ec40b4700 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.120+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651414351 0 0) 0x7f9ec413a050 con 0x7f9ec40a96b0 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.120+0000 7f9eb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.120+0000 7f9eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.120+0000 7f9eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.120+0000 7f9eb9ffb640 10 monclient(hunting): my global_id is 4160 2024-03-25T14:27:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.120+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec4135f10 con 0x7f9ec40a96b0 2024-03-25T14:27:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2026453010 0 0) 0x7f9ec4139d70 con 0x7f9ec40b4700 2024-03-25T14:27:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ec413a050 con 0x7f9ec40b4700 2024-03-25T14:27:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9ec00018d0 con 0x7f9ec40b4700 2024-03-25T14:27:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): dump: 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3043369087 0 0) 0x7f9ea8004eb0 con 0x7f9ec4142a20 2024-03-25T14:27:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ec4139d70 con 0x7f9ec4142a20 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9ebc001bd0 con 0x7f9ec4142a20 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.124+0000 7f9eb9ffb640 10 monclient(hunting): dump: 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.125+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088802033 0 0) 0x7f9ec4135f10 con 0x7f9ec40a96b0 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.125+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ea8004eb0 con 0x7f9ec40a96b0 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.125+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9eb4001ae0 con 0x7f9ec40a96b0 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.125+0000 7f9eb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.125+0000 7f9eb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.125+0000 7f9eb9ffb640 10 monclient(hunting): dump: 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:04.132 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:04.133 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3889024130 0 0) 0x7f9ec413a050 con 0x7f9ec40b4700 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 >> v1:172.21.15.26:6790/0 conn(0x7f9ec40a96b0 legacy=0x7f9ec4130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 >> v1:172.21.15.121:6789/0 conn(0x7f9ec4142a20 legacy=0x7f9ec41dba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 1 monclient: found mon.a 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec41363f0 con 0x7f9ec40b4700 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 10 monclient: _finish_auth 0 2024-03-25T14:27:04.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9eb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 5 monclient: authenticate success, global_id 4161 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 10 monclient: _renew_subs 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec4138d80 con 0x7f9ec40b4700 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 10 monclient: _renew_subs 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec4138670 con 0x7f9ec40b4700 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.128+0000 7f9ecb41e640 1 librados: init done 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.130+0000 7f9ecb41e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.130+0000 7f9ecb41e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.130+0000 7f9ecb41e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:04.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.130+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec4141220 con 0x7f9ec40b4700 2024-03-25T14:27:04.142 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:04.138+0000 7fded122b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T14:27:04.193 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:04.189+0000 7fea9aa38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T14:27:04.196 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:04.192+0000 7fded122b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T14:27:04.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ec0001d00 con 0x7f9ec40b4700 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9ec0004980 con 0x7f9ec40b4700 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 10 monclient: dump: 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:04.238 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9ebb7fe640 4 set_mon_vals no callback set 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9ec0004f30 con 0x7f9ec40b4700 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2669813156 0 0) 0x7f9ec00067b0 con 0x7f9ec40b4700 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 10 client.4161.objecter ms_dispatch 0x7f9ec4000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.233+0000 7f9eb9ffb640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.234+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9ec0006260 con 0x7f9ec40b4700 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.234+0000 7f9eb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:04.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.234+0000 7f9eb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:04.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.303+0000 7f9ecb41e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac269fe1-ef8f-4fe9-80e6-3081e6131db8", "id": 7}] 2024-03-25T14:27:04.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.303+0000 7f9ecb41e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac269fe1-ef8f-4fe9-80e6-3081e6131db8", "id": 7}] 2024-03-25T14:27:04.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.303+0000 7f9ecb41e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:04.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.303+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ac269fe1-ef8f-4fe9-80e6-3081e6131db8", "id": 7} v 0) v1 -- 0x7f9ec40b7420 con 0x7f9ec40b4700 2024-03-25T14:27:04.316 INFO:tasks.ceph.mgr.y.smithi026.stderr:2024-03-25T14:27:04.312+0000 7fded122b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T14:27:04.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.471+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 443849368 0 0) 0x7f9ec00051e0 con 0x7f9ec40b4700 2024-03-25T14:27:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.744+0000 7f9eb9ffb640 1 -- 172.21.15.121:0/2018994185 <== mon.0 v1:172.21.15.26:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac269fe1-ef8f-4fe9-80e6-3081e6131db8", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2730686977 0 1371665406) 0x7f9ec0015d60 con 0x7f9ec40b4700 2024-03-25T14:27:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.744+0000 7f9eb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac269fe1-ef8f-4fe9-80e6-3081e6131db8", "id": 7}] 2024-03-25T14:27:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.744+0000 7f9eb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:04.749 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 10 librados: watch_flush enter 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 10 librados: watch_flush exit 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 10 client.4161.objecter successfully canceled tick 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 10 monclient: shutdown 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 >> v1:172.21.15.26:6789/0 conn(0x7f9ec40b4700 legacy=0x7f9ec41db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 shutdown_connections 2024-03-25T14:27:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 >> 172.21.15.121:0/2018994185 conn(0x7f9ec40668e0 msgr2=0x7f9ec4130c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 shutdown_connections 2024-03-25T14:27:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 1 -- 172.21.15.121:0/2018994185 wait complete. 2024-03-25T14:27:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-25T14:27:04.745+0000 7f9ecb41e640 1 librados: shutdown 2024-03-25T14:27:04.760 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.760 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T14:27:04.760 DEBUG:teuthology.orchestra.run.smithi026:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T14:27:04.762 INFO:tasks.ceph.osd.0:Started 2024-03-25T14:27:04.762 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.763 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T14:27:04.763 DEBUG:teuthology.orchestra.run.smithi026:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T14:27:04.765 INFO:tasks.ceph.osd.1:Started 2024-03-25T14:27:04.765 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.765 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T14:27:04.765 DEBUG:teuthology.orchestra.run.smithi026:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T14:27:04.767 INFO:tasks.ceph.osd.2:Started 2024-03-25T14:27:04.768 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.768 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T14:27:04.768 DEBUG:teuthology.orchestra.run.smithi026:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T14:27:04.770 INFO:tasks.ceph.osd.3:Started 2024-03-25T14:27:04.770 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.770 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T14:27:04.770 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T14:27:04.803 INFO:tasks.ceph.osd.4:Started 2024-03-25T14:27:04.804 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.804 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T14:27:04.804 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T14:27:04.806 INFO:tasks.ceph.osd.5:Started 2024-03-25T14:27:04.806 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.806 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T14:27:04.806 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T14:27:04.808 INFO:tasks.ceph.osd.6:Started 2024-03-25T14:27:04.808 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:27:04.809 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T14:27:04.809 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T14:27:04.811 INFO:tasks.ceph.osd.7:Started 2024-03-25T14:27:04.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.956+0000 7f3309a93640 10 monclient: get_monmap_and_config 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.956+0000 7f3309a93640 10 monclient: build_initial_monmap 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.956+0000 7f3309a93640 10 monclient: monmap: 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:27:04.956990+0000 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:27:04.956990+0000 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:04.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:04.961 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:04.961 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 1 Processor -- start 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 1 -- start start 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient: init 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient(hunting): picked mon.noname-c con 0x7f33040b4700 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient(hunting): picked mon.noname-b con 0x7f3304142a20 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient(hunting): picked mon.noname-a con 0x7f33040a96b0 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33040a9d60 con 0x7f33040a96b0 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330413ff10 con 0x7f33040b4700 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33040aa1f0 con 0x7f3304142a20 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:04.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.957+0000 7f3309a93640 10 monclient(hunting): authenticate will time out at 1510.060373s 2024-03-25T14:27:04.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.974+0000 7f3302ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3304142a20 0x7f330413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51438/0 (socket says 172.21.15.26:51438) 2024-03-25T14:27:04.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:04.974+0000 7f3302ffd640 1 -- 172.21.15.26:0/3851618307 learned_addr learned my addr 172.21.15.26:0/3851618307 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.005+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716548183 0 0) 0x7f33040aa1f0 con 0x7f3304142a20 2024-03-25T14:27:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.005+0000 7f33027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.005+0000 7f33027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.005+0000 7f33027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.005+0000 7f33027fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T14:27:05.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.005+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ec003520 con 0x7f3304142a20 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421700621 0 0) 0x7f32ec003520 con 0x7f3304142a20 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 >> v1:172.21.15.121:6789/0 conn(0x7f33040b4700 legacy=0x7f3304142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 >> v1:172.21.15.26:6789/0 conn(0x7f33040a96b0 legacy=0x7f330413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 1 monclient: found mon.noname-b 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33040b4ae0 con 0x7f3304142a20 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f32f8001690 con 0x7f3304142a20 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f33027fc640 10 monclient: dump: 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:05.015 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f3309a93640 5 monclient: authenticate success, global_id 4163 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.010+0000 7f3309a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32f80017f0 con 0x7f3304142a20 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f33027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f33027fc640 1 -- 172.21.15.26:0/3851618307 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f32f8001c40 con 0x7f3304142a20 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f33027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f33027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f33027fc640 10 monclient: dump: 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:27:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f3309a93640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:05.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.012+0000 7f3309a93640 4 set_mon_vals no callback set 2024-03-25T14:27:05.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 10 monclient: shutdown 2024-03-25T14:27:05.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:05.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 1 -- 172.21.15.26:0/3851618307 >> v1:172.21.15.26:6790/0 conn(0x7f3304142a20 legacy=0x7f330413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:05.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 1 -- 172.21.15.26:0/3851618307 shutdown_connections 2024-03-25T14:27:05.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 1 -- 172.21.15.26:0/3851618307 >> 172.21.15.26:0/3851618307 conn(0x7f33040668e0 msgr2=0x7f3304066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 1 -- 172.21.15.26:0/3851618307 shutdown_connections 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.013+0000 7f3309a93640 1 -- 172.21.15.26:0/3851618307 wait complete. 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 10 monclient: build_initial_monmap 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 10 monclient: monmap: 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:05.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 1 librados: starting msgr at 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 1 librados: starting objecter 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 1 Processor -- start 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 1 -- start start 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 1 librados: setting wanted keys 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 1 librados: calling monclient init 2024-03-25T14:27:05.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.014+0000 7f3309a93640 10 monclient: init 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient(hunting): picked mon.noname-c con 0x7f33040a96b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient(hunting): picked mon.noname-a con 0x7f33040b4700 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient(hunting): picked mon.noname-b con 0x7f3304142a20 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33041372a0 con 0x7f33040b4700 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33041380b0 con 0x7f3304142a20 2024-03-25T14:27:05.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33041388e0 con 0x7f33040a96b0 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.015+0000 7f3309a93640 10 monclient(hunting): authenticate will time out at 1510.118087s 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.016+0000 7f33037fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f33040a96b0 0x7f330406ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51446/0 (socket says 172.21.15.26:51446) 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.016+0000 7f33037fe640 1 -- 172.21.15.26:0/1644456086 learned_addr learned my addr 172.21.15.26:0/1644456086 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.016+0000 7f32ebfff640 10 client.?.objecter ms_handle_connect 0x7f33040a96b0 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.016+0000 7f32ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.016+0000 7f32ebfff640 10 client.?.objecter ms_handle_connect 0x7f3304142a20 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.016+0000 7f32ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:05.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.018+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504699978 0 0) 0x7f33041388e0 con 0x7f33040a96b0 2024-03-25T14:27:05.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.018+0000 7f32ebfff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:05.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.018+0000 7f32ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:05.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.018+0000 7f32ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:05.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.018+0000 7f32ebfff640 10 monclient(hunting): my global_id is 4166 2024-03-25T14:27:05.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.018+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f0004eb0 con 0x7f33040a96b0 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826468157 0 0) 0x7f32f0004eb0 con 0x7f33040a96b0 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33041388e0 con 0x7f33040a96b0 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f32f40019d0 con 0x7f33040a96b0 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 10 monclient(hunting): dump: 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.028 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:27:05.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.024+0000 7f32ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:05.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3940522985 0 0) 0x7f33041388e0 con 0x7f33040a96b0 2024-03-25T14:27:05.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 >> v1:172.21.15.121:6789/0 conn(0x7f3304142a20 legacy=0x7f3304135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:05.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 >> v1:172.21.15.26:6789/0 conn(0x7f33040b4700 legacy=0x7f3304134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:05.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 1 monclient: found mon.c 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3304135d70 con 0x7f33040a96b0 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 10 monclient: _finish_auth 0 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f32ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f3309a93640 5 monclient: authenticate success, global_id 4166 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.028+0000 7f3309a93640 10 monclient: _renew_subs 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3304138380 con 0x7f33040a96b0 2024-03-25T14:27:05.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 10 monclient: _renew_subs 2024-03-25T14:27:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3304137730 con 0x7f33040a96b0 2024-03-25T14:27:05.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.029+0000 7f3309a93640 1 librados: init done 2024-03-25T14:27:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.030+0000 7f3309a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.030+0000 7f3309a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.030+0000 7f3309a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:05.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.030+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3304137030 con 0x7f33040a96b0 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32f4001c90 con 0x7f33040a96b0 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f32ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f32f4003a60 con 0x7f33040a96b0 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f32ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f32ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f32ebfff640 10 monclient: dump: 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-25T14:27:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.089+0000 7f3301ffb640 4 set_mon_vals no callback set 2024-03-25T14:27:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.090+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 443849368 0 0) 0x7f32f4003c80 con 0x7f33040a96b0 2024-03-25T14:27:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.093+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f32f40156b0 con 0x7f33040a96b0 2024-03-25T14:27:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.093+0000 7f32ebfff640 10 client.4166.objecter ms_dispatch 0x7f3304000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.093+0000 7f32ebfff640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.093+0000 7f32ebfff640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.093+0000 7f32ebfff640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-25T14:27:05.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.176+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f32f4014b50 con 0x7f33040a96b0 2024-03-25T14:27:05.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.176+0000 7f32ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:05.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.176+0000 7f32ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:05.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.245+0000 7f3309a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:05.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.245+0000 7f3309a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:05.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.245+0000 7f3309a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:05.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.245+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f330404f730 con 0x7f33040a96b0 2024-03-25T14:27:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.449+0000 7f32ebfff640 1 -- 172.21.15.26:0/1644456086 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f32f4013150 con 0x7f33040a96b0 2024-03-25T14:27:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.449+0000 7f32ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.449+0000 7f32ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:05.454 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-25T14:27:05.454 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 10 librados: watch_flush enter 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 10 librados: watch_flush exit 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 10 client.4166.objecter successfully canceled tick 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 10 monclient: shutdown 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 >> v1:172.21.15.26:6790/0 conn(0x7f33040a96b0 legacy=0x7f330406ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 shutdown_connections 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 >> 172.21.15.26:0/1644456086 conn(0x7f33040668e0 msgr2=0x7f330414e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 shutdown_connections 2024-03-25T14:27:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 1 -- 172.21.15.26:0/1644456086 wait complete. 2024-03-25T14:27:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-25T14:27:05.451+0000 7f3309a93640 1 librados: shutdown 2024-03-25T14:27:05.471 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T14:27:05.471 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T14:27:05.821 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:05.817+0000 7fea3b6dd640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T14:27:05.859 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-25T14:27:05.855+0000 7fea3b6dd640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T14:27:06.277 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:06.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.423+0000 7f2a3bd60640 10 monclient: get_monmap_and_config 2024-03-25T14:27:06.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.423+0000 7f2a3bd60640 10 monclient: build_initial_monmap 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.423+0000 7f2a3bd60640 10 monclient: monmap: 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:06.424359+0000 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:06.424359+0000 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:06.428 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.424+0000 7f2a3bd60640 1 Processor -- start 2024-03-25T14:27:06.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.425+0000 7f2a3bd60640 1 -- start start 2024-03-25T14:27:06.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.425+0000 7f2a3bd60640 10 monclient: init 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.426+0000 7f2a3bd60640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.426+0000 7f2a3bd60640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.426+0000 7f2a3bd60640 10 monclient(hunting): picked mon.noname-a con 0x7f2a340b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.426+0000 7f2a3bd60640 10 monclient(hunting): picked mon.noname-c con 0x7f2a340b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a3bd60640 10 monclient(hunting): picked mon.noname-b con 0x7f2a3414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a3bd60640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340af940 con 0x7f2a340b4d60 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a3bd60640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340b0a70 con 0x7f2a340b1730 2024-03-25T14:27:06.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a3bd60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a340b0d50 con 0x7f2a3414be90 2024-03-25T14:27:06.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a3bd60640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:06.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a3bd60640 10 monclient(hunting): authenticate will time out at 1511.530154s 2024-03-25T14:27:06.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a392fb640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2a340b1730 0x7f2a340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:50648/0 (socket says 172.21.15.26:50648) 2024-03-25T14:27:06.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.427+0000 7f2a392fb640 1 -- 172.21.15.26:0/1117430334 learned_addr learned my addr 172.21.15.26:0/1117430334 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:06.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.432+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653936023 0 0) 0x7f2a340b0a70 con 0x7f2a340b1730 2024-03-25T14:27:06.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.432+0000 7f2a38afa640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.432+0000 7f2a38afa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.432+0000 7f2a38afa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.432+0000 7f2a38afa640 10 monclient(hunting): my global_id is 4171 2024-03-25T14:27:06.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.432+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a20003520 con 0x7f2a340b1730 2024-03-25T14:27:06.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.434+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946830909 0 0) 0x7f2a340af940 con 0x7f2a340b4d60 2024-03-25T14:27:06.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.434+0000 7f2a38afa640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.434+0000 7f2a38afa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.434+0000 7f2a38afa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.434+0000 7f2a38afa640 10 monclient(hunting): my global_id is 4182 2024-03-25T14:27:06.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.434+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a340b0a70 con 0x7f2a340b4d60 2024-03-25T14:27:06.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777078762 0 0) 0x7f2a20003520 con 0x7f2a340b1730 2024-03-25T14:27:06.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 >> v1:172.21.15.26:6790/0 conn(0x7f2a3414be90 legacy=0x7f2a3414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:27:06.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 >> v1:172.21.15.26:6789/0 conn(0x7f2a340b4d60 legacy=0x7f2a3413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: _finish_hunting 0 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 1 monclient: found mon.noname-c 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a340b1030 con 0x7f2a340b1730 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: _finish_auth 0 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: _check_auth_tickets 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a3bd60640 5 monclient: authenticate success, global_id 4171 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a3bd60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a24001690 con 0x7f2a340b1730 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a38afa640 10 monclient: dump: 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.437+0000 7f2a3bd60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a24001870 con 0x7f2a340b1730 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a38afa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a38afa640 1 -- 172.21.15.26:0/1117430334 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a24001c80 con 0x7f2a340b1730 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a38afa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a38afa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a38afa640 10 monclient: dump: 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a3bd60640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.440+0000 7f2a3bd60640 4 set_mon_vals no callback set 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.441+0000 7f2a3bd60640 10 monclient: shutdown 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.441+0000 7f2a3bd60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.441+0000 7f2a3bd60640 1 -- 172.21.15.26:0/1117430334 >> v1:172.21.15.121:6789/0 conn(0x7f2a340b1730 legacy=0x7f2a340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.441+0000 7f2a3bd60640 1 -- 172.21.15.26:0/1117430334 shutdown_connections 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.441+0000 7f2a3bd60640 1 -- 172.21.15.26:0/1117430334 >> 172.21.15.26:0/1117430334 conn(0x7f2a340617a0 msgr2=0x7f2a34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.442+0000 7f2a3bd60640 1 -- 172.21.15.26:0/1117430334 shutdown_connections 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.442+0000 7f2a3bd60640 1 -- 172.21.15.26:0/1117430334 wait complete. 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 10 monclient: build_initial_monmap 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 10 monclient: monmap: 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:06.448 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:06.449 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:06.449 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:06.449 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:06.449 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:06.449 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:06.449 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 1 librados: starting msgr at 2024-03-25T14:27:06.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 1 librados: starting objecter 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 1 Processor -- start 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 1 -- start start 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.443+0000 7f2a3bd60640 1 librados: setting wanted keys 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.444+0000 7f2a3bd60640 1 librados: calling monclient init 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.444+0000 7f2a3bd60640 10 monclient: init 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 10 monclient(hunting): picked mon.noname-b con 0x7f2a340b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 10 monclient(hunting): picked mon.noname-c con 0x7f2a340b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 10 monclient(hunting): picked mon.noname-a con 0x7f2a3414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a341d78f0 con 0x7f2a3414be90 2024-03-25T14:27:06.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a341d8490 con 0x7f2a340b1730 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a341d8620 con 0x7f2a340b4d60 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.445+0000 7f2a3bd60640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.446+0000 7f2a3bd60640 10 monclient(hunting): authenticate will time out at 1511.548650s 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.446+0000 7f2a39afc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2a340b1730 0x7f2a341d7d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:50662/0 (socket says 172.21.15.26:50662) 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.446+0000 7f2a39afc640 1 -- 172.21.15.26:0/4239345502 learned_addr learned my addr 172.21.15.26:0/4239345502 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.447+0000 7f2a2a7fc640 10 client.?.objecter ms_handle_connect 0x7f2a340b1730 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.447+0000 7f2a2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.447+0000 7f2a2a7fc640 10 client.?.objecter ms_handle_connect 0x7f2a3414be90 2024-03-25T14:27:06.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.447+0000 7f2a2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:06.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989277979 0 0) 0x7f2a341d8490 con 0x7f2a340b1730 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 10 monclient(hunting): my global_id is 4174 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a18004eb0 con 0x7f2a340b1730 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 10 client.?.objecter ms_handle_connect 0x7f2a340b4d60 2024-03-25T14:27:06.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.450+0000 7f2a2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:06.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.453+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881524412 0 0) 0x7f2a341d8620 con 0x7f2a340b4d60 2024-03-25T14:27:06.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.453+0000 7f2a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.453+0000 7f2a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.454+0000 7f2a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.454+0000 7f2a2a7fc640 10 monclient(hunting): my global_id is 4172 2024-03-25T14:27:06.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.454+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a341d8490 con 0x7f2a340b4d60 2024-03-25T14:27:06.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.454+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363135971 0 0) 0x7f2a341d78f0 con 0x7f2a3414be90 2024-03-25T14:27:06.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.454+0000 7f2a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.455+0000 7f2a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.455+0000 7f2a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.455+0000 7f2a2a7fc640 10 monclient(hunting): my global_id is 4185 2024-03-25T14:27:06.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.455+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a341d8620 con 0x7f2a3414be90 2024-03-25T14:27:06.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.456+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2298273059 0 0) 0x7f2a18004eb0 con 0x7f2a340b1730 2024-03-25T14:27:06.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.457+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a341d78f0 con 0x7f2a340b1730 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.458+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a30001ab0 con 0x7f2a340b1730 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.458+0000 7f2a2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.458+0000 7f2a2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.458+0000 7f2a2a7fc640 10 monclient(hunting): dump: 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:06.462 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:06.463 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.458+0000 7f2a2a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:06.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.460+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736074369 0 0) 0x7f2a341d8490 con 0x7f2a340b4d60 2024-03-25T14:27:06.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.460+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a18004eb0 con 0x7f2a340b4d60 2024-03-25T14:27:06.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.460+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a24001b80 con 0x7f2a340b4d60 2024-03-25T14:27:06.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.460+0000 7f2a2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.460+0000 7f2a2a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:06.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.460+0000 7f2a2a7fc640 10 monclient(hunting): dump: 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:06.465 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1644475817 0 0) 0x7f2a341d78f0 con 0x7f2a340b1730 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 >> v1:172.21.15.26:6790/0 conn(0x7f2a340b4d60 legacy=0x7f2a3413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 >> v1:172.21.15.26:6789/0 conn(0x7f2a3414be90 legacy=0x7f2a34137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 1 monclient: found mon.b 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3413a970 con 0x7f2a340b1730 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a3bd60640 5 monclient: authenticate success, global_id 4174 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a3bd60640 10 monclient: _renew_subs 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a3bd60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a34143900 con 0x7f2a340b1730 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.462+0000 7f2a3bd60640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.463+0000 7f2a3bd60640 10 monclient: _renew_subs 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.463+0000 7f2a3bd60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.463+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a341397c0 con 0x7f2a340b1730 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.463+0000 7f2a3bd60640 1 librados: init done 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.464+0000 7f2a3bd60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.464+0000 7f2a3bd60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.464+0000 7f2a3bd60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.464+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3413abc0 con 0x7f2a340b1730 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.465+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a30001c10 con 0x7f2a340b1730 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.465+0000 7f2a2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:06.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.465+0000 7f2a2bfff640 4 set_mon_vals no callback set 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.908+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a300038c0 con 0x7f2a340b1730 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.909+0000 7f2a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.909+0000 7f2a2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.909+0000 7f2a2a7fc640 10 monclient: dump: 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:06.913 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:06.914 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:06.914 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:06.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.910+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 1781593023 0 0) 0x7f2a30003aa0 con 0x7f2a340b1730 2024-03-25T14:27:06.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.910+0000 7f2a392fb640 10 monclient: get_auth_request con 0x7f2a1806ddf0 auth_method 0 2024-03-25T14:27:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.911+0000 7f2a2a7fc640 10 client.4174.objecter ms_handle_connect 0x7f2a1806ddf0 2024-03-25T14:27:06.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.913+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f2a30054770 con 0x7f2a340b1730 2024-03-25T14:27:06.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.913+0000 7f2a2a7fc640 10 client.4174.objecter ms_dispatch 0x7f2a34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:06.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.913+0000 7f2a2a7fc640 3 client.4174.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:06.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.913+0000 7f2a2a7fc640 3 client.4174.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:06.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.913+0000 7f2a2a7fc640 20 client.4174.objecter dump_active .. 0 homeless 2024-03-25T14:27:06.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.917+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a30050280 con 0x7f2a340b1730 2024-03-25T14:27:06.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.917+0000 7f2a2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:06.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:06.917+0000 7f2a2a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:07.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.255+0000 7f2a3bd60640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:07.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.255+0000 7f2a3bd60640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:07.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.255+0000 7f2a3bd60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:07.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.255+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a3413bab0 con 0x7f2a340b1730 2024-03-25T14:27:07.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.330+0000 7f2a2a7fc640 1 -- 172.21.15.26:0/4239345502 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f2a300521b0 con 0x7f2a340b1730 2024-03-25T14:27:07.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.330+0000 7f2a2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:07.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.330+0000 7f2a2a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:07.335 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:07.335 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 10 librados: watch_flush enter 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 10 librados: watch_flush exit 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 20 client.4174.objecter shutdown clearing up homeless session... 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 10 client.4174.objecter successfully canceled tick 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 >> v1:172.21.15.121:6800/288813066 conn(0x7f2a1806ddf0 legacy=0x7f2a180702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 10 monclient: shutdown 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 >> v1:172.21.15.121:6789/0 conn(0x7f2a340b1730 legacy=0x7f2a341d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 shutdown_connections 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 >> 172.21.15.26:0/4239345502 conn(0x7f2a340617a0 msgr2=0x7f2a3413c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 shutdown_connections 2024-03-25T14:27:07.338 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.333+0000 7f2a3bd60640 1 -- 172.21.15.26:0/4239345502 wait complete. 2024-03-25T14:27:07.339 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:07.334+0000 7f2a3bd60640 1 librados: shutdown 2024-03-25T14:27:07.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:11.623 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:06.750 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:11.686 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:06.813 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:11.734 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:06.863 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:11.757 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:06.889 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:11.794 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:06.924 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:11.818 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:06.950 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:11.968 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:07.101 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.026 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:07.159 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.133 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:07.297 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.180 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:07.326 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.196 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:07.360 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.229 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-03-25T14:27:12.219+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.234 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:07.379 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.241 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:07.388 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.246 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:07.391 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.304 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:07.449 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.306 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:07.451 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.375 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-03-25T14:27:12.365+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.380 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-03-25T14:27:12.369+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.574 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-03-25T14:27:12.563+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.691 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:07.819 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.744 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-03-25T14:27:12.733+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.796 INFO:tasks.ceph.osd.3.smithi026.stderr:2024-03-25T14:27:12.785+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.838 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:07.970 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.840 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:07.969 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:12.857 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-03-25T14:27:12.846+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:12.902 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-03-25T14:27:12.891+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:27:13.035 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:08.168 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.058 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:08.186 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.209 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:08.341 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.213 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:08.343 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.224 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:08.388 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.256 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:08.402 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.338 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:08.483 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.376 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:08.520 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.381 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:08.509 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.398 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:08.532 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.532 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:08.661 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.569 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:08.701 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.593 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:08.757 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.629 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:08.776 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.646 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:08.773 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.707 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:08.852 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.713 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:08.845 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.754 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:08.898 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.796 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:08.926 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.857 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:08.988 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.908 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:09.071 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.910 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:09.037 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.947 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:09.094 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:13.979 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:09.112 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.060 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:09.190 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.068 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:09.213 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.114 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:09.258 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.143 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:09.275 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.160 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:14.182 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:09.346 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.187 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:09.315 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.226 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:09.372 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.243 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:09.376 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.244+0000 7f24da68f640 10 monclient: get_monmap_and_config 2024-03-25T14:27:14.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.244+0000 7f24da68f640 10 monclient: build_initial_monmap 2024-03-25T14:27:14.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.244+0000 7f24da68f640 10 monclient: monmap: 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:14.246763+0000 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:14.246763+0000 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:14.250 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.245+0000 7f24da68f640 1 Processor -- start 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.245+0000 7f24da68f640 1 -- start start 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.245+0000 7f24da68f640 10 monclient: init 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient(hunting): picked mon.noname-c con 0x7f24d40b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:14.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient(hunting): picked mon.noname-a con 0x7f24d40b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient(hunting): picked mon.noname-b con 0x7f24d414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40af940 con 0x7f24d40b1730 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40b0a70 con 0x7f24d40b4d60 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40b0d50 con 0x7f24d414be90 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24da68f640 10 monclient(hunting): authenticate will time out at 1519.349960s 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24d8c2c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f24d414be90 0x7f24d414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47086/0 (socket says 172.21.15.26:47086) 2024-03-25T14:27:14.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.246+0000 7f24d8c2c640 1 -- 172.21.15.26:0/1239300781 learned_addr learned my addr 172.21.15.26:0/1239300781 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:14.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.249+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179936507 0 0) 0x7f24d40b0a70 con 0x7f24d40b4d60 2024-03-25T14:27:14.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.249+0000 7f24d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.249+0000 7f24d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.249+0000 7f24d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.249+0000 7f24d2ffd640 10 monclient(hunting): my global_id is 4201 2024-03-25T14:27:14.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.249+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24bc003520 con 0x7f24d40b4d60 2024-03-25T14:27:14.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78443606 0 0) 0x7f24d40af940 con 0x7f24d40b1730 2024-03-25T14:27:14.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d40b0a70 con 0x7f24d40b1730 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479467961 0 0) 0x7f24d40b0d50 con 0x7f24d414be90 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 10 monclient(hunting): my global_id is 4199 2024-03-25T14:27:14.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.250+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d40af940 con 0x7f24d414be90 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 805236672 0 0) 0x7f24bc003520 con 0x7f24d40b4d60 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 >> v1:172.21.15.26:6790/0 conn(0x7f24d414be90 legacy=0x7f24d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 >> v1:172.21.15.26:6789/0 conn(0x7f24d40b1730 legacy=0x7f24d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 1 monclient: found mon.noname-c 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:27:14.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b1030 con 0x7f24d40b4d60 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 10 monclient: _finish_auth 0 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f24c4001690 con 0x7f24d40b4d60 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.252+0000 7f24d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.253+0000 7f24d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.253+0000 7f24d2ffd640 10 monclient: dump: 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.259 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.253+0000 7f24da68f640 5 monclient: authenticate success, global_id 4201 2024-03-25T14:27:14.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.253+0000 7f24da68f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:14.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24c40017f0 con 0x7f24d40b4d60 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24d2ffd640 1 -- 172.21.15.26:0/1239300781 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f24c4001c00 con 0x7f24d40b4d60 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24d2ffd640 10 monclient: dump: 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.264 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24da68f640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.254+0000 7f24da68f640 4 set_mon_vals no callback set 2024-03-25T14:27:14.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 10 monclient: shutdown 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 1 -- 172.21.15.26:0/1239300781 >> v1:172.21.15.121:6789/0 conn(0x7f24d40b4d60 legacy=0x7f24d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 1 -- 172.21.15.26:0/1239300781 shutdown_connections 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 1 -- 172.21.15.26:0/1239300781 >> 172.21.15.26:0/1239300781 conn(0x7f24d40617a0 msgr2=0x7f24d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 1 -- 172.21.15.26:0/1239300781 shutdown_connections 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.255+0000 7f24da68f640 1 -- 172.21.15.26:0/1239300781 wait complete. 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 10 monclient: build_initial_monmap 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 10 monclient: monmap: 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:14.266 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 1 librados: starting msgr at 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 1 librados: starting objecter 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 1 Processor -- start 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 1 -- start start 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 1 librados: setting wanted keys 2024-03-25T14:27:14.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 1 librados: calling monclient init 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.256+0000 7f24da68f640 10 monclient: init 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient(hunting): picked mon.noname-b con 0x7f24d40b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient(hunting): picked mon.noname-a con 0x7f24d40b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient(hunting): picked mon.noname-c con 0x7f24d414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4142050 con 0x7f24d40b4d60 2024-03-25T14:27:14.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d41dac50 con 0x7f24d40b1730 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4140d00 con 0x7f24d414be90 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24da68f640 10 monclient(hunting): authenticate will time out at 1519.361120s 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.257+0000 7f24d37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f24d40b4d60 0x7f24d41db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49552/0 (socket says 172.21.15.26:49552) 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.258+0000 7f24d37fe640 1 -- 172.21.15.26:0/790291364 learned_addr learned my addr 172.21.15.26:0/790291364 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.258+0000 7f24d0ff9640 10 client.?.objecter ms_handle_connect 0x7f24d40b4d60 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.258+0000 7f24d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.258+0000 7f24d0ff9640 10 client.?.objecter ms_handle_connect 0x7f24d40b1730 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.258+0000 7f24d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.259+0000 7f24d0ff9640 10 client.?.objecter ms_handle_connect 0x7f24d414be90 2024-03-25T14:27:14.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.259+0000 7f24d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:14.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.260+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669842199 0 0) 0x7f24d41dac50 con 0x7f24d40b1730 2024-03-25T14:27:14.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.260+0000 7f24d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.260+0000 7f24d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.260+0000 7f24d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.260+0000 7f24d0ff9640 10 monclient(hunting): my global_id is 4204 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.260+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c0004eb0 con 0x7f24d40b1730 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.261+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867073604 0 0) 0x7f24d4140d00 con 0x7f24d414be90 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.261+0000 7f24d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.261+0000 7f24d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.261+0000 7f24d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.261+0000 7f24d0ff9640 10 monclient(hunting): my global_id is 4202 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.261+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d41dac50 con 0x7f24d414be90 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474164561 0 0) 0x7f24c0004eb0 con 0x7f24d40b1730 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d4140d00 con 0x7f24d40b1730 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f24c4001b00 con 0x7f24d40b1730 2024-03-25T14:27:14.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 10 monclient(hunting): dump: 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:14.273 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.265+0000 7f24d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.266+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191057798 0 0) 0x7f24d41dac50 con 0x7f24d414be90 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.266+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c0004eb0 con 0x7f24d414be90 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.266+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f24cc001b50 con 0x7f24d414be90 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.266+0000 7f24d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.266+0000 7f24d0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.266+0000 7f24d0ff9640 10 monclient(hunting): dump: 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.274 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.275 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:14.275 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:14.275 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:14.275 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:14.275 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1797901705 0 0) 0x7f24d4140d00 con 0x7f24d40b1730 2024-03-25T14:27:14.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 >> v1:172.21.15.26:6790/0 conn(0x7f24d414be90 legacy=0x7f24d41e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 >> v1:172.21.15.26:6789/0 conn(0x7f24d40b4d60 legacy=0x7f24d41db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:27:14.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 1 monclient: found mon.b 2024-03-25T14:27:14.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d41423e0 con 0x7f24d40b1730 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 10 monclient: _finish_auth 0 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 5 monclient: authenticate success, global_id 4204 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 10 monclient: _renew_subs 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d413b470 con 0x7f24d40b1730 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:14.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 10 monclient: _renew_subs 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d41dba40 con 0x7f24d40b1730 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.268+0000 7f24da68f640 1 librados: init done 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.269+0000 7f24da68f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.269+0000 7f24da68f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.269+0000 7f24da68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:14.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.269+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d41401f0 con 0x7f24d40b1730 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24c4001220 con 0x7f24d40b1730 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f24c4003a40 con 0x7f24d40b1730 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d0ff9640 10 monclient: dump: 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:14.288 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:14.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.283+0000 7f24d27fc640 4 set_mon_vals no callback set 2024-03-25T14:27:14.290 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.284+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f24c4019a10 con 0x7f24d40b1730 2024-03-25T14:27:14.290 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.285+0000 7f24d37fe640 10 monclient: get_auth_request con 0x7f24c006df10 auth_method 0 2024-03-25T14:27:14.290 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.285+0000 7f24d0ff9640 10 client.4204.objecter ms_handle_connect 0x7f24c006df10 2024-03-25T14:27:14.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.286+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f24c4053640 con 0x7f24d40b1730 2024-03-25T14:27:14.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.286+0000 7f24d0ff9640 10 client.4204.objecter ms_dispatch 0x7f24d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:14.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.286+0000 7f24d0ff9640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:14.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.286+0000 7f24d0ff9640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:14.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.286+0000 7f24d0ff9640 20 client.4204.objecter dump_active .. 0 homeless 2024-03-25T14:27:14.326 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:09.454 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.357 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:09.502 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.409 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:09.553 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.434 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:09.566 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.446 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:09.610 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.464 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:09.591 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.490 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:09.637 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.515 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:09.648 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.528+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24c404f150 con 0x7f24d40b1730 2024-03-25T14:27:14.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.528+0000 7f24d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:14.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.528+0000 7f24d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:14.593 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:09.723 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.647 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:09.792 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.701 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:09.845 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.712 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:09.876 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.718+0000 7f24da68f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:14.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.718+0000 7f24da68f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:14.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.718+0000 7f24da68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:14.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.718+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24d4140e90 con 0x7f24d40b1730 2024-03-25T14:27:14.731 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:09.858 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.732+0000 7f24d0ff9640 1 -- 172.21.15.26:0/790291364 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f24c4051080 con 0x7f24d40b1730 2024-03-25T14:27:14.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.732+0000 7f24d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:14.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.732+0000 7f24d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:14.737 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:14.737 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:14.737 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:09.870 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.739 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 10 librados: watch_flush enter 2024-03-25T14:27:14.739 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 10 librados: watch_flush exit 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 20 client.4204.objecter shutdown clearing up homeless session... 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 10 client.4204.objecter successfully canceled tick 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 >> v1:172.21.15.121:6800/288813066 conn(0x7f24c006df10 legacy=0x7f24c00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 10 monclient: shutdown 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 >> v1:172.21.15.121:6789/0 conn(0x7f24d40b1730 legacy=0x7f24d41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 shutdown_connections 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 >> 172.21.15.26:0/790291364 conn(0x7f24d40617a0 msgr2=0x7f24d40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 shutdown_connections 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 -- 172.21.15.26:0/790291364 wait complete. 2024-03-25T14:27:14.740 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:14.734+0000 7f24da68f640 1 librados: shutdown 2024-03-25T14:27:14.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:14.760 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:09.907 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.788 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:09.921 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.857 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:09.987 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.936 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:10.081 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:14.978 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:10.142 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.007 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:10.151 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.007 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:10.134 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.016 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:10.148 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.039 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:10.185 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.066 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:10.200 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.130 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:10.260 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.225 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:10.370 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.247 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:10.411 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.299 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:10.443 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.306 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:10.438 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.307 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:10.453 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.337 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:10.470 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.486 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:10.614 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.514 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:10.678 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.520 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:10.665 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.539 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:10.667 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.575 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:10.721 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.598 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:10.742 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.600 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:10.730 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.655 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:10.785 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.816 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:10.961 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.854 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:10.985 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.854 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:10.987 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.890 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:11.034 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.908 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:11.040 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.954 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:11.087 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:15.993 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:11.157 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.049 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:11.212 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.052 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:11.198 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.108 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:11.254 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.362 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:11.507 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.418 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:11.563 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.419 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:11.564 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:16.476 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:11.620 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:20.974 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:16.104 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.060 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:16.187 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.223 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:16.355 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.257 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:16.390 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.555 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:21.572 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:16.736 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.642+0000 7f679fa9a640 10 monclient: get_monmap_and_config 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.642+0000 7f679fa9a640 10 monclient: build_initial_monmap 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.642+0000 7f679fa9a640 10 monclient: monmap: 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:21.644785+0000 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:21.644785+0000 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:21.648 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.643+0000 7f679fa9a640 1 Processor -- start 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.643+0000 7f679fa9a640 1 -- start start 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.643+0000 7f679fa9a640 10 monclient: init 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient(hunting): picked mon.noname-c con 0x7f67980b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient(hunting): picked mon.noname-a con 0x7f67980b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient(hunting): picked mon.noname-b con 0x7f679814be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67980af940 con 0x7f67980b1730 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67980b0a70 con 0x7f67980b4d60 2024-03-25T14:27:21.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67980b0d50 con 0x7f679814be90 2024-03-25T14:27:21.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:21.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.644+0000 7f679fa9a640 10 monclient(hunting): authenticate will time out at 1526.748048s 2024-03-25T14:27:21.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.645+0000 7f679d035640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67980b1730 0x7f67980b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39018/0 (socket says 172.21.15.26:39018) 2024-03-25T14:27:21.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.645+0000 7f679d035640 1 -- 172.21.15.26:0/2812892548 learned_addr learned my addr 172.21.15.26:0/2812892548 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:21.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237252840 0 0) 0x7f67980af940 con 0x7f67980b1730 2024-03-25T14:27:21.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): my global_id is 4218 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6784003520 con 0x7f67980b1730 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293926001 0 0) 0x7f67980b0d50 con 0x7f679814be90 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): my global_id is 4205 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67980af940 con 0x7f679814be90 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839981303 0 0) 0x7f67980b0a70 con 0x7f67980b4d60 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 10 monclient(hunting): my global_id is 4207 2024-03-25T14:27:21.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.648+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67980b0d50 con 0x7f67980b4d60 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802868112 0 0) 0x7f67980af940 con 0x7f679814be90 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 >> v1:172.21.15.121:6789/0 conn(0x7f67980b4d60 legacy=0x7f679813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 >> v1:172.21.15.26:6789/0 conn(0x7f67980b1730 legacy=0x7f67980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: _finish_hunting 0 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 1 monclient: found mon.noname-b 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67980b1030 con 0x7f679814be90 2024-03-25T14:27:21.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: _finish_auth 0 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: _check_auth_tickets 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f67900019f0 con 0x7f679814be90 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679c834640 10 monclient: dump: 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:21.658 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:21.659 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:21.659 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:21.659 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:21.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679fa9a640 5 monclient: authenticate success, global_id 4205 2024-03-25T14:27:21.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.652+0000 7f679fa9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:21.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6790001b50 con 0x7f679814be90 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679c834640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679c834640 1 -- 172.21.15.26:0/2812892548 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6790001f60 con 0x7f679814be90 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679c834640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679c834640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679c834640 10 monclient: dump: 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:21.660 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:21.661 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:21.661 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:21.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679fa9a640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:21.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679fa9a640 4 set_mon_vals no callback set 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679fa9a640 10 monclient: shutdown 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679fa9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.655+0000 7f679fa9a640 1 -- 172.21.15.26:0/2812892548 >> v1:172.21.15.26:6790/0 conn(0x7f679814be90 legacy=0x7f679814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.656+0000 7f679fa9a640 1 -- 172.21.15.26:0/2812892548 shutdown_connections 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.656+0000 7f679fa9a640 1 -- 172.21.15.26:0/2812892548 >> 172.21.15.26:0/2812892548 conn(0x7f67980617a0 msgr2=0x7f6798061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.656+0000 7f679fa9a640 1 -- 172.21.15.26:0/2812892548 shutdown_connections 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.656+0000 7f679fa9a640 1 -- 172.21.15.26:0/2812892548 wait complete. 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 10 monclient: build_initial_monmap 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 10 monclient: monmap: 2024-03-25T14:27:21.663 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 1 librados: starting msgr at 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 1 librados: starting objecter 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 1 Processor -- start 2024-03-25T14:27:21.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 1 -- start start 2024-03-25T14:27:21.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 1 librados: setting wanted keys 2024-03-25T14:27:21.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 1 librados: calling monclient init 2024-03-25T14:27:21.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.657+0000 7f679fa9a640 10 monclient: init 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient(hunting): picked mon.noname-c con 0x7f67980b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient(hunting): picked mon.noname-a con 0x7f67980b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient(hunting): picked mon.noname-b con 0x7f679814be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67981384b0 con 0x7f67980b4d60 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67981327e0 con 0x7f679814be90 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6798137e70 con 0x7f67980b1730 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.658+0000 7f679fa9a640 10 monclient(hunting): authenticate will time out at 1526.762573s 2024-03-25T14:27:21.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.659+0000 7f679e037640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f679814be90 0x7f67981320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:35240/0 (socket says 172.21.15.26:35240) 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.659+0000 7f679e037640 1 -- 172.21.15.26:0/1545950774 learned_addr learned my addr 172.21.15.26:0/1545950774 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.659+0000 7f678e7fc640 10 client.?.objecter ms_handle_connect 0x7f679814be90 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.659+0000 7f678e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.660+0000 7f678e7fc640 10 client.?.objecter ms_handle_connect 0x7f67980b4d60 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.660+0000 7f678e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.660+0000 7f678e7fc640 10 client.?.objecter ms_handle_connect 0x7f67980b1730 2024-03-25T14:27:21.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.660+0000 7f678e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:21.668 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:16.811 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.661+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235848743 0 0) 0x7f67981327e0 con 0x7f679814be90 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.661+0000 7f678e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.661+0000 7f678e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.661+0000 7f678e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.661+0000 7f678e7fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.661+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f677c004eb0 con 0x7f679814be90 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148476042 0 0) 0x7f6798137e70 con 0x7f67980b1730 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): my global_id is 4208 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67981327e0 con 0x7f67980b1730 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095581719 0 0) 0x7f67981384b0 con 0x7f67980b4d60 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 10 monclient(hunting): my global_id is 4221 2024-03-25T14:27:21.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.663+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798137e70 con 0x7f67980b4d60 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.665+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840278952 0 0) 0x7f677c004eb0 con 0x7f679814be90 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.665+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67981384b0 con 0x7f679814be90 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.665+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6790001fa0 con 0x7f679814be90 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.666+0000 7f678e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.666+0000 7f678e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.666+0000 7f678e7fc640 10 monclient(hunting): dump: 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:21.672 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:21.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.666+0000 7f678e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3508252413 0 0) 0x7f67981384b0 con 0x7f679814be90 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 >> v1:172.21.15.26:6790/0 conn(0x7f67980b1730 legacy=0x7f67981dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 >> v1:172.21.15.26:6789/0 conn(0x7f67980b4d60 legacy=0x7f67981319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 1 monclient: found mon.b 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6798138000 con 0x7f679814be90 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f678e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 5 monclient: authenticate success, global_id 4210 2024-03-25T14:27:21.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 10 monclient: _renew_subs 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6798145330 con 0x7f679814be90 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 10 monclient: _renew_subs 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6798140380 con 0x7f679814be90 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.670+0000 7f679fa9a640 1 librados: init done 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.672+0000 7f679fa9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.672+0000 7f679fa9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.672+0000 7f679fa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:21.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:21.672+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67980627b0 con 0x7f679814be90 2024-03-25T14:27:21.769 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:16.914 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:21.945 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:17.089 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6790003c20 con 0x7f679814be90 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6790003ff0 con 0x7f679814be90 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678e7fc640 10 monclient: dump: 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:22.019 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:22.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.015+0000 7f678ffff640 4 set_mon_vals no callback set 2024-03-25T14:27:22.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.016+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f6790004210 con 0x7f679814be90 2024-03-25T14:27:22.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.017+0000 7f679d836640 10 monclient: get_auth_request con 0x7f677c06ddf0 auth_method 0 2024-03-25T14:27:22.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.017+0000 7f678e7fc640 10 client.4210.objecter ms_handle_connect 0x7f677c06ddf0 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f6790054ac0 con 0x7f679814be90 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 10 client.4210.objecter ms_dispatch 0x7f6798000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67900505d0 con 0x7f679814be90 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:22.025 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.020+0000 7f678e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:22.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.322+0000 7f679fa9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:22.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.322+0000 7f679fa9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:22.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.322+0000 7f679fa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:22.328 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.322+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6798138660 con 0x7f679814be90 2024-03-25T14:27:22.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.340+0000 7f678e7fc640 1 -- 172.21.15.26:0/1545950774 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f6798138660 con 0x7f679814be90 2024-03-25T14:27:22.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.340+0000 7f678e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:22.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.340+0000 7f678e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:22.344 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:22.344 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:22.347 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 10 librados: watch_flush enter 2024-03-25T14:27:22.347 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 10 librados: watch_flush exit 2024-03-25T14:27:22.347 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T14:27:22.347 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 10 client.4210.objecter successfully canceled tick 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 >> v1:172.21.15.121:6800/288813066 conn(0x7f677c06ddf0 legacy=0x7f677c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 10 monclient: shutdown 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.342+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 >> v1:172.21.15.121:6789/0 conn(0x7f679814be90 legacy=0x7f67981320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.343+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 shutdown_connections 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.343+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 >> 172.21.15.26:0/1545950774 conn(0x7f67980617a0 msgr2=0x7f679813c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.343+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 shutdown_connections 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.343+0000 7f679fa9a640 1 -- 172.21.15.26:0/1545950774 wait complete. 2024-03-25T14:27:22.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:22.343+0000 7f679fa9a640 1 librados: shutdown 2024-03-25T14:27:22.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:26.751 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:21.881 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:26.755 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:21.884 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:26.762 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:21.889 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:26.766 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:21.893 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.085 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:22.216 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.088 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:22.220 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.108 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:22.272 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.112 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:22.276 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.141 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:22.274 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.147 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:22.280 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.274 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:22.418 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.278 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:22.422 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.305 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:22.451 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.309 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:22.455 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.490 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:22.635 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:27.497 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:22.642 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:29.166 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.289+0000 7efc220e4640 10 monclient: get_monmap_and_config 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.289+0000 7efc220e4640 10 monclient: build_initial_monmap 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.289+0000 7efc220e4640 10 monclient: monmap: 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:29.289910+0000 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:29.289910+0000 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:29.293 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:29.294 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:29.294 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.289+0000 7efc220e4640 1 Processor -- start 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.289+0000 7efc220e4640 1 -- start start 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.289+0000 7efc220e4640 10 monclient: init 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient(hunting): picked mon.noname-b con 0x7efc1c0b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient(hunting): picked mon.noname-a con 0x7efc1c0b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient(hunting): picked mon.noname-c con 0x7efc1c14be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0af940 con 0x7efc1c0b1730 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0b0a70 con 0x7efc1c14be90 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0b0d50 con 0x7efc1c0b4d60 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:29.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.290+0000 7efc220e4640 10 monclient(hunting): authenticate will time out at 1534.393373s 2024-03-25T14:27:29.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.291+0000 7efc1b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efc1c0b4d60 0x7efc1c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53544/0 (socket says 172.21.15.26:53544) 2024-03-25T14:27:29.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.291+0000 7efc1b7fe640 1 -- 172.21.15.26:0/940322582 learned_addr learned my addr 172.21.15.26:0/940322582 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:29.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.291+0000 7efc1bfff640 1 --1- 172.21.15.26:0/940322582 >> v1:172.21.15.121:6789/0 conn(0x7efc1c14be90 0x7efc1c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:59474/0 (socket says 172.21.15.26:59474) 2024-03-25T14:27:29.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541076475 0 0) 0x7efc1c0af940 con 0x7efc1c0b1730 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc04003520 con 0x7efc1c0b1730 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220345333 0 0) 0x7efc1c0b0a70 con 0x7efc1c14be90 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c0af940 con 0x7efc1c14be90 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093707756 0 0) 0x7efc1c0b0d50 con 0x7efc1c0b4d60 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T14:27:29.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.294+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c0b0a70 con 0x7efc1c0b4d60 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120254204 0 0) 0x7efc04003520 con 0x7efc1c0b1730 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 >> v1:172.21.15.26:6790/0 conn(0x7efc1c0b4d60 legacy=0x7efc1c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 >> v1:172.21.15.121:6789/0 conn(0x7efc1c14be90 legacy=0x7efc1c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 1 monclient: found mon.noname-a 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0b1030 con 0x7efc1c0b1730 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:29.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc10001690 con 0x7efc1c0b1730 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc1a7fc640 10 monclient: dump: 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc220e4640 5 monclient: authenticate success, global_id 4224 2024-03-25T14:27:29.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.298+0000 7efc220e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:29.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc100017f0 con 0x7efc1c0b1730 2024-03-25T14:27:29.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:29.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc1a7fc640 1 -- 172.21.15.26:0/940322582 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc10001c00 con 0x7efc1c0b1730 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc1a7fc640 10 monclient: dump: 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc220e4640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:29.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.300+0000 7efc220e4640 4 set_mon_vals no callback set 2024-03-25T14:27:29.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 10 monclient: shutdown 2024-03-25T14:27:29.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:29.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 1 -- 172.21.15.26:0/940322582 >> v1:172.21.15.26:6789/0 conn(0x7efc1c0b1730 legacy=0x7efc1c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 1 -- 172.21.15.26:0/940322582 shutdown_connections 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 1 -- 172.21.15.26:0/940322582 >> 172.21.15.26:0/940322582 conn(0x7efc1c0617a0 msgr2=0x7efc1c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 1 -- 172.21.15.26:0/940322582 shutdown_connections 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.301+0000 7efc220e4640 1 -- 172.21.15.26:0/940322582 wait complete. 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.302+0000 7efc220e4640 10 monclient: build_initial_monmap 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.302+0000 7efc220e4640 10 monclient: monmap: 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:29.307 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 1 librados: starting msgr at 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 1 librados: starting objecter 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 1 Processor -- start 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 1 -- start start 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 1 librados: setting wanted keys 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 1 librados: calling monclient init 2024-03-25T14:27:29.308 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.303+0000 7efc220e4640 10 monclient: init 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.304+0000 7efc220e4640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.304+0000 7efc220e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.304+0000 7efc220e4640 10 monclient(hunting): picked mon.noname-a con 0x7efc1c0b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.304+0000 7efc220e4640 10 monclient(hunting): picked mon.noname-b con 0x7efc1c0b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc1bfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efc1c14be90 0x7efc1c1e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53550/0 (socket says 172.21.15.26:53550) 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc1bfff640 1 -- 172.21.15.26:0/197170226 learned_addr learned my addr 172.21.15.26:0/197170226 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc1affd640 1 --1- 172.21.15.26:0/197170226 >> v1:172.21.15.121:6789/0 conn(0x7efc1c0b4d60 0x7efc1c144f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:59484/0 (socket says 172.21.15.26:59484) 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc220e4640 10 monclient(hunting): picked mon.noname-c con 0x7efc1c14be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c144d40 con 0x7efc1c0b1730 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c1dbac0 con 0x7efc1c0b4d60 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c135d30 con 0x7efc1c14be90 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc220e4640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc220e4640 10 monclient(hunting): authenticate will time out at 1534.407853s 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc03fff640 10 client.?.objecter ms_handle_connect 0x7efc1c0b1730 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc03fff640 10 client.?.objecter ms_handle_connect 0x7efc1c14be90 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc03fff640 10 client.?.objecter ms_handle_connect 0x7efc1c0b4d60 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.305+0000 7efc03fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.307+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223001497 0 0) 0x7efc1c135d30 con 0x7efc1c14be90 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.307+0000 7efc03fff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.307+0000 7efc03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.307+0000 7efc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.307+0000 7efc03fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.307+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc08004eb0 con 0x7efc1c14be90 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601884898 0 0) 0x7efc1c1dbac0 con 0x7efc1c0b4d60 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): my global_id is 4216 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c135d30 con 0x7efc1c0b4d60 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573892154 0 0) 0x7efc1c144d40 con 0x7efc1c0b1730 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 10 monclient(hunting): my global_id is 4227 2024-03-25T14:27:29.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.308+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c1dbac0 con 0x7efc1c0b1730 2024-03-25T14:27:29.317 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 974918401 0 0) 0x7efc08004eb0 con 0x7efc1c14be90 2024-03-25T14:27:29.317 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc1c144d40 con 0x7efc1c14be90 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc14001c70 con 0x7efc1c14be90 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 10 monclient(hunting): dump: 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:29.318 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:29.319 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.319 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.313+0000 7efc03fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:29.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.314+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655682567 0 0) 0x7efc1c135d30 con 0x7efc1c0b4d60 2024-03-25T14:27:29.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.314+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc08004eb0 con 0x7efc1c0b4d60 2024-03-25T14:27:29.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.314+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc10001b00 con 0x7efc1c0b4d60 2024-03-25T14:27:29.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.314+0000 7efc03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.314+0000 7efc03fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:29.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.314+0000 7efc03fff640 10 monclient(hunting): dump: 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.315+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030394090 0 0) 0x7efc1c1dbac0 con 0x7efc1c0b1730 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.315+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc1c135d30 con 0x7efc1c0b1730 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.315+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc0c001830 con 0x7efc1c0b1730 2024-03-25T14:27:29.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.315+0000 7efc03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.315+0000 7efc03fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.315+0000 7efc03fff640 10 monclient(hunting): dump: 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:29.322 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108673682 0 0) 0x7efc1c144d40 con 0x7efc1c14be90 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 >> v1:172.21.15.121:6789/0 conn(0x7efc1c0b4d60 legacy=0x7efc1c144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 >> v1:172.21.15.26:6789/0 conn(0x7efc1c0b1730 legacy=0x7efc1c1db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 1 monclient: found mon.c 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:29.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c1382f0 con 0x7efc1c14be90 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 10 monclient: _finish_auth 0 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 5 monclient: authenticate success, global_id 4214 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 10 monclient: _renew_subs 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c145670 con 0x7efc1c14be90 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 10 monclient: _renew_subs 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c145920 con 0x7efc1c14be90 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.317+0000 7efc220e4640 1 librados: init done 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.319+0000 7efc220e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.319+0000 7efc220e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:29.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.319+0000 7efc220e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:29.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.319+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c1448e0 con 0x7efc1c14be90 2024-03-25T14:27:29.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc14004b60 con 0x7efc1c14be90 2024-03-25T14:27:29.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc19ffb640 4 set_mon_vals no callback set 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efc14004cd0 con 0x7efc1c14be90 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.517+0000 7efc03fff640 10 monclient: dump: 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:29.522 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:29.523 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:29.523 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:29.523 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:29.523 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.518+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7efc1401aca0 con 0x7efc1c14be90 2024-03-25T14:27:29.523 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.519+0000 7efc1b7fe640 10 monclient: get_auth_request con 0x7efc0806e140 auth_method 0 2024-03-25T14:27:29.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.519+0000 7efc03fff640 10 client.4214.objecter ms_handle_connect 0x7efc0806e140 2024-03-25T14:27:29.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.627+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7efc14054860 con 0x7efc1c14be90 2024-03-25T14:27:29.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.627+0000 7efc03fff640 10 client.4214.objecter ms_dispatch 0x7efc1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:29.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.627+0000 7efc03fff640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:29.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.627+0000 7efc03fff640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:29.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.627+0000 7efc03fff640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T14:27:29.727 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.723+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc14050370 con 0x7efc1c14be90 2024-03-25T14:27:29.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.723+0000 7efc03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:29.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.723+0000 7efc03fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:29.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.949+0000 7efc220e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:29.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.949+0000 7efc220e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:29.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.949+0000 7efc220e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:27:29.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.949+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc1c0627b0 con 0x7efc1c14be90 2024-03-25T14:27:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.965+0000 7efc03fff640 1 -- 172.21.15.26:0/197170226 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7efc140522a0 con 0x7efc1c14be90 2024-03-25T14:27:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.965+0000 7efc03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:29.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.965+0000 7efc03fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:29.970 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:29.970 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.967+0000 7efc220e4640 10 librados: watch_flush enter 2024-03-25T14:27:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 10 librados: watch_flush exit 2024-03-25T14:27:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T14:27:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 10 client.4214.objecter successfully canceled tick 2024-03-25T14:27:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 >> v1:172.21.15.121:6800/288813066 conn(0x7efc0806e140 legacy=0x7efc08070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 10 monclient: shutdown 2024-03-25T14:27:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 >> v1:172.21.15.26:6790/0 conn(0x7efc1c14be90 legacy=0x7efc1c1e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 shutdown_connections 2024-03-25T14:27:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 >> 172.21.15.26:0/197170226 conn(0x7efc1c0617a0 msgr2=0x7efc1c0b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 shutdown_connections 2024-03-25T14:27:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 -- 172.21.15.26:0/197170226 wait complete. 2024-03-25T14:27:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:29.968+0000 7efc220e4640 1 librados: shutdown 2024-03-25T14:27:29.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:31.008 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:00:26.138 35320== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.101 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:00:26.228 35321== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.388 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:00:26.520 35319== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.525 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:00:26.689 35702== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.578 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:00:26.724 35704== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.582 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:00:26.726 35706== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.795 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:00:26.928 35318== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:31.946 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:00:27.091 35705== Warning: unimplemented fcntl command: 1038 2024-03-25T14:27:36.793 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:36.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.917+0000 7fb22cd48640 10 monclient: get_monmap_and_config 2024-03-25T14:27:36.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.917+0000 7fb22cd48640 10 monclient: build_initial_monmap 2024-03-25T14:27:36.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.917+0000 7fb22cd48640 10 monclient: monmap: 2024-03-25T14:27:36.921 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:36.921 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:36.917916+0000 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:36.917916+0000 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.917+0000 7fb22cd48640 1 Processor -- start 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.917+0000 7fb22cd48640 1 -- start start 2024-03-25T14:27:36.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.917+0000 7fb22cd48640 10 monclient: init 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 10 monclient(hunting): picked mon.noname-a con 0x7fb2280a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 10 monclient(hunting): picked mon.noname-b con 0x7fb22813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 10 monclient(hunting): picked mon.noname-c con 0x7fb2281430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb228142520 con 0x7fb2280a98f0 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2280b28a0 con 0x7fb2281430c0 2024-03-25T14:27:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.918+0000 7fb22cd48640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2280b2a30 con 0x7fb22813f710 2024-03-25T14:27:36.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.919+0000 7fb22cd48640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:36.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.919+0000 7fb22cd48640 10 monclient(hunting): authenticate will time out at 1542.021605s 2024-03-25T14:27:36.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.919+0000 7fb225d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb22813f710 0x7fb2281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53554/0 (socket says 172.21.15.26:53554) 2024-03-25T14:27:36.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.919+0000 7fb225d9b640 1 -- 172.21.15.26:0/2693522674 learned_addr learned my addr 172.21.15.26:0/2693522674 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:36.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.993+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114955690 0 0) 0x7fb228142520 con 0x7fb2280a98f0 2024-03-25T14:27:36.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.993+0000 7fb22559a640 10 monclient(hunting): _negotiate 2024-03-25T14:27:36.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.993+0000 7fb22559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:36.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.993+0000 7fb22559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:36.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.993+0000 7fb22559a640 10 monclient(hunting): my global_id is 4230 2024-03-25T14:27:36.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.993+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb214003520 con 0x7fb2280a98f0 2024-03-25T14:27:37.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987922182 0 0) 0x7fb214003520 con 0x7fb2280a98f0 2024-03-25T14:27:37.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 >> v1:172.21.15.26:6790/0 conn(0x7fb22813f710 legacy=0x7fb2281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 >> v1:172.21.15.121:6789/0 conn(0x7fb2281430c0 legacy=0x7fb22814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: _finish_hunting 0 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 1 monclient: found mon.noname-a 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2280b44c0 con 0x7fb2280a98f0 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: _finish_auth 0 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: _check_auth_tickets 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb210001690 con 0x7fb2280a98f0 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22559a640 10 monclient: dump: 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:37.003 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22cd48640 5 monclient: authenticate success, global_id 4230 2024-03-25T14:27:37.004 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:36.998+0000 7fb22cd48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:37.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2100017f0 con 0x7fb2280a98f0 2024-03-25T14:27:37.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:37.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22559a640 1 -- 172.21.15.26:0/2693522674 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb210001c40 con 0x7fb2280a98f0 2024-03-25T14:27:37.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22559a640 10 monclient: dump: 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:37.007 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:37.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22cd48640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:37.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.001+0000 7fb22cd48640 4 set_mon_vals no callback set 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.002+0000 7fb22cd48640 10 monclient: shutdown 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.002+0000 7fb22cd48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.002+0000 7fb22cd48640 1 -- 172.21.15.26:0/2693522674 >> v1:172.21.15.26:6789/0 conn(0x7fb2280a98f0 legacy=0x7fb2280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.003+0000 7fb22cd48640 1 -- 172.21.15.26:0/2693522674 shutdown_connections 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.003+0000 7fb22cd48640 1 -- 172.21.15.26:0/2693522674 >> 172.21.15.26:0/2693522674 conn(0x7fb2280b7ea0 msgr2=0x7fb2280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.003+0000 7fb22cd48640 1 -- 172.21.15.26:0/2693522674 shutdown_connections 2024-03-25T14:27:37.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.003+0000 7fb22cd48640 1 -- 172.21.15.26:0/2693522674 wait complete. 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.004+0000 7fb22cd48640 10 monclient: build_initial_monmap 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.005+0000 7fb22cd48640 10 monclient: monmap: 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:37.010 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:37.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.005+0000 7fb22cd48640 1 librados: starting msgr at 2024-03-25T14:27:37.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.005+0000 7fb22cd48640 1 librados: starting objecter 2024-03-25T14:27:37.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.005+0000 7fb22cd48640 1 Processor -- start 2024-03-25T14:27:37.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.006+0000 7fb22cd48640 1 -- start start 2024-03-25T14:27:37.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.006+0000 7fb22cd48640 1 librados: setting wanted keys 2024-03-25T14:27:37.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.006+0000 7fb22cd48640 1 librados: calling monclient init 2024-03-25T14:27:37.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.006+0000 7fb22cd48640 10 monclient: init 2024-03-25T14:27:37.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.007+0000 7fb22cd48640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.007+0000 7fb22cd48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.007+0000 7fb22cd48640 10 monclient(hunting): picked mon.noname-a con 0x7fb2280a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.007+0000 7fb22cd48640 10 monclient(hunting): picked mon.noname-c con 0x7fb22813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.007+0000 7fb22cd48640 10 monclient(hunting): picked mon.noname-b con 0x7fb2281430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.007+0000 7fb22cd48640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb228135ac0 con 0x7fb2280a98f0 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb22cd48640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb228137030 con 0x7fb2281430c0 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb22cd48640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2281372a0 con 0x7fb22813f710 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb22cd48640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb22659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb2280a98f0 0x7fb228133440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38138/0 (socket says 172.21.15.26:38138) 2024-03-25T14:27:37.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb22659c640 1 -- 172.21.15.26:0/1020650785 learned_addr learned my addr 172.21.15.26:0/1020650785 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:37.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb22cd48640 10 monclient(hunting): authenticate will time out at 1542.111125s 2024-03-25T14:27:37.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb21effd640 10 client.?.objecter ms_handle_connect 0x7fb2280a98f0 2024-03-25T14:27:37.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.008+0000 7fb21effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.014+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225652817 0 0) 0x7fb228135ac0 con 0x7fb2280a98f0 2024-03-25T14:27:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.014+0000 7fb21effd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.014+0000 7fb21effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.014+0000 7fb21effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.014+0000 7fb21effd640 10 monclient(hunting): my global_id is 4233 2024-03-25T14:27:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.014+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb208004eb0 con 0x7fb2280a98f0 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.022+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3282087879 0 0) 0x7fb208004eb0 con 0x7fb2280a98f0 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.022+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb228135ac0 con 0x7fb2280a98f0 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.022+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb210001690 con 0x7fb2280a98f0 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.022+0000 7fb21effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.023+0000 7fb21effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.023+0000 7fb21effd640 10 monclient(hunting): dump: 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.023+0000 7fb21effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662720615 0 0) 0x7fb228135ac0 con 0x7fb2280a98f0 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 >> v1:172.21.15.26:6790/0 conn(0x7fb22813f710 legacy=0x7fb2281379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 >> v1:172.21.15.121:6789/0 conn(0x7fb2281430c0 legacy=0x7fb2281e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 10 monclient: _finish_hunting 0 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 1 monclient: found mon.a 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb228137580 con 0x7fb2280a98f0 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 10 monclient: _finish_auth 0 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 10 monclient: _check_auth_tickets 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb21effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 5 monclient: authenticate success, global_id 4233 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 10 monclient: _renew_subs 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2281363b0 con 0x7fb2280a98f0 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:37.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 10 monclient: _renew_subs 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2281353a0 con 0x7fb2280a98f0 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.028+0000 7fb22cd48640 1 librados: init done 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.030+0000 7fb22cd48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.030+0000 7fb22cd48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.030+0000 7fb22cd48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:37.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.030+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2280627b0 con 0x7fb2280a98f0 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb210003960 con 0x7fb2280a98f0 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb210003db0 con 0x7fb2280a98f0 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 10 monclient: dump: 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.053 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb224d99640 4 set_mon_vals no callback set 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.048+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fb210003fd0 con 0x7fb2280a98f0 2024-03-25T14:27:37.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.050+0000 7fb225d9b640 10 monclient: get_auth_request con 0x7fb20806dc50 auth_method 0 2024-03-25T14:27:37.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.050+0000 7fb21effd640 10 client.4233.objecter ms_handle_connect 0x7fb20806dc50 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.053+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fb210054bd0 con 0x7fb2280a98f0 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.053+0000 7fb21effd640 10 client.4233.objecter ms_dispatch 0x7fb228000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.053+0000 7fb21effd640 3 client.4233.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.053+0000 7fb21effd640 3 client.4233.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.053+0000 7fb21effd640 20 client.4233.objecter dump_active .. 0 homeless 2024-03-25T14:27:37.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.301+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2100506e0 con 0x7fb2280a98f0 2024-03-25T14:27:37.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.301+0000 7fb21effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:37.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.301+0000 7fb21effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:37.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.658+0000 7fb22cd48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:37.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.658+0000 7fb22cd48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:37.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.658+0000 7fb22cd48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:27:37.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.658+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb22813e1f0 con 0x7fb2280a98f0 2024-03-25T14:27:37.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.726+0000 7fb21effd640 1 -- 172.21.15.26:0/1020650785 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fb210052610 con 0x7fb2280a98f0 2024-03-25T14:27:37.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.726+0000 7fb21effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:37.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.726+0000 7fb21effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:37.731 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:37.731 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:37.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.729+0000 7fb22cd48640 10 librados: watch_flush enter 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.729+0000 7fb22cd48640 10 librados: watch_flush exit 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 20 client.4233.objecter shutdown clearing up homeless session... 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 10 client.4233.objecter successfully canceled tick 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 >> v1:172.21.15.121:6800/288813066 conn(0x7fb20806dc50 legacy=0x7fb208070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 10 monclient: shutdown 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 >> v1:172.21.15.26:6789/0 conn(0x7fb2280a98f0 legacy=0x7fb228133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 shutdown_connections 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 >> 172.21.15.26:0/1020650785 conn(0x7fb2280b7ea0 msgr2=0x7fb228074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:37.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 shutdown_connections 2024-03-25T14:27:37.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 -- 172.21.15.26:0/1020650785 wait complete. 2024-03-25T14:27:37.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:37.730+0000 7fb22cd48640 1 librados: shutdown 2024-03-25T14:27:37.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:38.946 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-03-25T14:27:38.939+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T14:27:39.241 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-03-25T14:27:39.233+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T14:27:39.766 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-03-25T14:27:39.759+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T14:27:39.912 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-03-25T14:27:39.905+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T14:27:40.236 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-03-25T14:27:40.229+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T14:27:40.264 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-03-25T14:27:40.257+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T14:27:40.341 INFO:tasks.ceph.osd.3.smithi026.stderr:2024-03-25T14:27:40.334+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T14:27:41.028 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-03-25T14:27:41.022+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T14:27:44.556 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:44.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.663+0000 7f296fd4d640 10 monclient: get_monmap_and_config 2024-03-25T14:27:44.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.663+0000 7f296fd4d640 10 monclient: build_initial_monmap 2024-03-25T14:27:44.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.663+0000 7f296fd4d640 10 monclient: monmap: 2024-03-25T14:27:44.667 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:44.664112+0000 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:44.664112+0000 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:44.668 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:44.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.664+0000 7f296fd4d640 1 Processor -- start 2024-03-25T14:27:44.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.664+0000 7f296fd4d640 1 -- start start 2024-03-25T14:27:44.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.664+0000 7f296fd4d640 10 monclient: init 2024-03-25T14:27:44.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f29680a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f296813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f29681430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2968142520 con 0x7f296813f710 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680b28a0 con 0x7f29680a98f0 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680b2a30 con 0x7f29681430c0 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:44.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.665+0000 7f296fd4d640 10 monclient(hunting): authenticate will time out at 1549.767858s 2024-03-25T14:27:44.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.676+0000 7f296dae9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f29680a98f0 0x7f29680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42086/0 (socket says 172.21.15.26:42086) 2024-03-25T14:27:44.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.676+0000 7f296dae9640 1 -- 172.21.15.26:0/3134945365 learned_addr learned my addr 172.21.15.26:0/3134945365 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:44.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.689+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133603124 0 0) 0x7f29680b28a0 con 0x7f29680a98f0 2024-03-25T14:27:44.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.689+0000 7f296cae7640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.689+0000 7f296cae7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.689+0000 7f296cae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.689+0000 7f296cae7640 10 monclient(hunting): my global_id is 4243 2024-03-25T14:27:44.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.689+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2954003520 con 0x7f29680a98f0 2024-03-25T14:27:44.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.696+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048687016 0 0) 0x7f29680b2a30 con 0x7f29681430c0 2024-03-25T14:27:44.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.696+0000 7f296cae7640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.696+0000 7f296cae7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.696+0000 7f296cae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.696+0000 7f296cae7640 10 monclient(hunting): my global_id is 4241 2024-03-25T14:27:44.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.696+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29680b28a0 con 0x7f29681430c0 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.700+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591398615 0 0) 0x7f2968142520 con 0x7f296813f710 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.700+0000 7f296cae7640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.700+0000 7f296cae7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.700+0000 7f296cae7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.700+0000 7f296cae7640 10 monclient(hunting): my global_id is 4260 2024-03-25T14:27:44.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.700+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29680b2a30 con 0x7f296813f710 2024-03-25T14:27:44.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952674773 0 0) 0x7f29680b28a0 con 0x7f29681430c0 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 >> v1:172.21.15.121:6789/0 conn(0x7f29680a98f0 legacy=0x7f29680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 >> v1:172.21.15.26:6789/0 conn(0x7f296813f710 legacy=0x7f29681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: _finish_hunting 0 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 1 monclient: found mon.noname-b 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29680b44c0 con 0x7f29681430c0 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: _finish_auth 0 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: _check_auth_tickets 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2960001670 con 0x7f29681430c0 2024-03-25T14:27:44.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296cae7640 10 monclient: dump: 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:44.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296fd4d640 5 monclient: authenticate success, global_id 4241 2024-03-25T14:27:44.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.704+0000 7f296fd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:44.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29600017d0 con 0x7f29681430c0 2024-03-25T14:27:44.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296cae7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296cae7640 1 -- 172.21.15.26:0/3134945365 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2960001be0 con 0x7f29681430c0 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296fd4d640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296fd4d640 4 set_mon_vals no callback set 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296cae7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296cae7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.714+0000 7f296cae7640 10 monclient: dump: 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:44.720 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:44.720 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:44.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 10 monclient: shutdown 2024-03-25T14:27:44.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:44.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 1 -- 172.21.15.26:0/3134945365 >> v1:172.21.15.26:6790/0 conn(0x7f29681430c0 legacy=0x7f296814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 1 -- 172.21.15.26:0/3134945365 shutdown_connections 2024-03-25T14:27:44.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 1 -- 172.21.15.26:0/3134945365 >> 172.21.15.26:0/3134945365 conn(0x7f29680b7ea0 msgr2=0x7f29680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 1 -- 172.21.15.26:0/3134945365 shutdown_connections 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.715+0000 7f296fd4d640 1 -- 172.21.15.26:0/3134945365 wait complete. 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.716+0000 7f296fd4d640 10 monclient: build_initial_monmap 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.716+0000 7f296fd4d640 10 monclient: monmap: 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:44.722 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 1 librados: starting msgr at 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 1 librados: starting objecter 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 1 Processor -- start 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 1 -- start start 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 1 librados: setting wanted keys 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 1 librados: calling monclient init 2024-03-25T14:27:44.723 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.717+0000 7f296fd4d640 10 monclient: init 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f29680a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f296813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f29681430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29681d73f0 con 0x7f29680a98f0 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29681395e0 con 0x7f29681430c0 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29681398c0 con 0x7f296813f710 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.718+0000 7f296fd4d640 10 monclient(hunting): authenticate will time out at 1549.820811s 2024-03-25T14:27:44.726 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.722+0000 7f296e2ea640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f29681430c0 0x7f2968137fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42142/0 (socket says 172.21.15.26:42142) 2024-03-25T14:27:44.726 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.722+0000 7f296e2ea640 1 -- 172.21.15.26:0/2160747661 learned_addr learned my addr 172.21.15.26:0/2160747661 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:44.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.724+0000 7f295e7fc640 10 client.?.objecter ms_handle_connect 0x7f29681430c0 2024-03-25T14:27:44.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.724+0000 7f295e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:44.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.731+0000 7f295e7fc640 10 client.?.objecter ms_handle_connect 0x7f296813f710 2024-03-25T14:27:44.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.731+0000 7f295e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:44.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.732+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689096492 0 0) 0x7f29681395e0 con 0x7f29681430c0 2024-03-25T14:27:44.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.732+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.732+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.732+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.732+0000 7f295e7fc640 10 monclient(hunting): my global_id is 4246 2024-03-25T14:27:44.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.732+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c004eb0 con 0x7f29681430c0 2024-03-25T14:27:44.744 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.740+0000 7f295e7fc640 10 client.?.objecter ms_handle_connect 0x7f29680a98f0 2024-03-25T14:27:44.744 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.740+0000 7f295e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:44.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.748+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606235409 0 0) 0x7f29681398c0 con 0x7f296813f710 2024-03-25T14:27:44.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.748+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.748+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.748+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.748+0000 7f295e7fc640 10 monclient(hunting): my global_id is 4244 2024-03-25T14:27:44.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.748+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29681395e0 con 0x7f296813f710 2024-03-25T14:27:44.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.762+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198501288 0 0) 0x7f29681d73f0 con 0x7f29680a98f0 2024-03-25T14:27:44.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.762+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.762+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.762+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.762+0000 7f295e7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T14:27:44.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.762+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29681398c0 con 0x7f29680a98f0 2024-03-25T14:27:44.768 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 917582816 0 0) 0x7f294c004eb0 con 0x7f29681430c0 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29681d73f0 con 0x7f29681430c0 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2960001c20 con 0x7f29681430c0 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:44.769 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:44.770 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:44.770 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:44.770 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.764+0000 7f295e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.772+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2252920292 0 0) 0x7f29681395e0 con 0x7f296813f710 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.772+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f294c004eb0 con 0x7f296813f710 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.772+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2958001870 con 0x7f296813f710 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.772+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.772+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.772+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.777 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:44.778 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.778 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:44.778 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:44.778 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:44.778 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:44.778 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:44.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607184365 0 0) 0x7f29681d73f0 con 0x7f29681430c0 2024-03-25T14:27:44.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 >> v1:172.21.15.26:6790/0 conn(0x7f296813f710 legacy=0x7f29681d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 >> v1:172.21.15.26:6789/0 conn(0x7f29680a98f0 legacy=0x7f29681d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 1 monclient: found mon.b 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2968137b40 con 0x7f29681430c0 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f295e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 5 monclient: authenticate success, global_id 4246 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 10 monclient: _renew_subs 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29681d7f90 con 0x7f29681430c0 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:44.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 10 monclient: _renew_subs 2024-03-25T14:27:44.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:44.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29681d4b60 con 0x7f29681430c0 2024-03-25T14:27:44.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.784+0000 7f296fd4d640 1 librados: init done 2024-03-25T14:27:44.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.785+0000 7f296fd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:44.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.785+0000 7f296fd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:44.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.785+0000 7f296fd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:44.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:44.785+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29680627b0 con 0x7f29681430c0 2024-03-25T14:27:45.088 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29600038a0 con 0x7f29681430c0 2024-03-25T14:27:45.088 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2960003bf0 con 0x7f29681430c0 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295e7fc640 10 monclient: dump: 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:45.089 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:45.090 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:45.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.084+0000 7f295ffff640 4 set_mon_vals no callback set 2024-03-25T14:27:45.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.085+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f2960003e10 con 0x7f29681430c0 2024-03-25T14:27:45.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.086+0000 7f296dae9640 10 monclient: get_auth_request con 0x7f294c06df70 auth_method 0 2024-03-25T14:27:45.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.086+0000 7f295e7fc640 10 client.4246.objecter ms_handle_connect 0x7f294c06df70 2024-03-25T14:27:45.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f29600547e0 con 0x7f29681430c0 2024-03-25T14:27:45.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 10 client.4246.objecter ms_dispatch 0x7f2968000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:45.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:45.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:45.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-25T14:27:45.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29600502f0 con 0x7f29681430c0 2024-03-25T14:27:45.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:45.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.089+0000 7f295e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:45.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.337+0000 7f296fd4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:45.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.337+0000 7f296fd4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:45.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.337+0000 7f296fd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:45.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.337+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2968141a00 con 0x7f29681430c0 2024-03-25T14:27:45.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.489+0000 7f295e7fc640 1 -- 172.21.15.26:0/2160747661 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f2960052220 con 0x7f29681430c0 2024-03-25T14:27:45.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.489+0000 7f295e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:45.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.489+0000 7f295e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:45.494 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:45.494 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 10 librados: watch_flush enter 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 10 librados: watch_flush exit 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 10 client.4246.objecter successfully canceled tick 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 >> v1:172.21.15.121:6800/288813066 conn(0x7f294c06df70 legacy=0x7f294c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 10 monclient: shutdown 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 >> v1:172.21.15.121:6789/0 conn(0x7f29681430c0 legacy=0x7f2968137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:45.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 shutdown_connections 2024-03-25T14:27:45.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 >> 172.21.15.26:0/2160747661 conn(0x7f29680b7ea0 msgr2=0x7f2968074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:45.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.491+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 shutdown_connections 2024-03-25T14:27:45.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.492+0000 7f296fd4d640 1 -- 172.21.15.26:0/2160747661 wait complete. 2024-03-25T14:27:45.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:45.492+0000 7f296fd4d640 1 librados: shutdown 2024-03-25T14:27:45.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:52.315 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.428+0000 7f9378825640 10 monclient: get_monmap_and_config 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.428+0000 7f9378825640 10 monclient: build_initial_monmap 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.428+0000 7f9378825640 10 monclient: monmap: 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:27:52.428900+0000 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:27:52.428900+0000 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:52.432 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:52.433 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:27:52.433 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:27:52.433 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.428+0000 7f9378825640 1 Processor -- start 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.428+0000 7f9378825640 1 -- start start 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.428+0000 7f9378825640 10 monclient: init 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient(hunting): picked mon.noname-b con 0x7f93740b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient(hunting): picked mon.noname-a con 0x7f93740b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient(hunting): picked mon.noname-c con 0x7f937414be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:52.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93740af940 con 0x7f93740b1730 2024-03-25T14:27:52.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93740b0a70 con 0x7f937414be90 2024-03-25T14:27:52.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93740b0d50 con 0x7f93740b4d60 2024-03-25T14:27:52.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:52.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.429+0000 7f9378825640 10 monclient(hunting): authenticate will time out at 1557.532562s 2024-03-25T14:27:52.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.439+0000 7f9372d9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f937414be90 0x7f937414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60882/0 (socket says 172.21.15.26:60882) 2024-03-25T14:27:52.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.439+0000 7f9372d9d640 1 -- 172.21.15.26:0/3381778075 learned_addr learned my addr 172.21.15.26:0/3381778075 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:52.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.448+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874639613 0 0) 0x7f93740af940 con 0x7f93740b1730 2024-03-25T14:27:52.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.448+0000 7f937159a640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.448+0000 7f937159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.448+0000 7f937159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.448+0000 7f937159a640 10 monclient(hunting): my global_id is 4266 2024-03-25T14:27:52.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.448+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c003520 con 0x7f93740b1730 2024-03-25T14:27:52.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.454+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953897808 0 0) 0x7f93740b0d50 con 0x7f93740b4d60 2024-03-25T14:27:52.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.454+0000 7f937159a640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.454+0000 7f937159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.454+0000 7f937159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.454+0000 7f937159a640 10 monclient(hunting): my global_id is 4247 2024-03-25T14:27:52.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.454+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93740af940 con 0x7f93740b4d60 2024-03-25T14:27:52.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.456+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346606779 0 0) 0x7f93740b0a70 con 0x7f937414be90 2024-03-25T14:27:52.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.456+0000 7f937159a640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.456+0000 7f937159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.456+0000 7f937159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.456+0000 7f937159a640 10 monclient(hunting): my global_id is 4249 2024-03-25T14:27:52.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.456+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c0039d0 con 0x7f937414be90 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.463+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007979328 0 0) 0x7f935c003520 con 0x7f93740b1730 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.463+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 >> v1:172.21.15.26:6790/0 conn(0x7f93740b4d60 legacy=0x7f937413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.463+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 >> v1:172.21.15.121:6789/0 conn(0x7f937414be90 legacy=0x7f937414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.463+0000 7f937159a640 10 monclient: _finish_hunting 0 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.463+0000 7f937159a640 1 monclient: found mon.noname-a 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.463+0000 7f937159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740b1030 con 0x7f93740b1730 2024-03-25T14:27:52.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 10 monclient: _finish_auth 0 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 10 monclient: _check_auth_tickets 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9368001690 con 0x7f93740b1730 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f937159a640 10 monclient: dump: 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:52.469 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:52.470 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:52.470 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:52.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f9378825640 5 monclient: authenticate success, global_id 4266 2024-03-25T14:27:52.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.464+0000 7f9378825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:52.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93680017f0 con 0x7f93740b1730 2024-03-25T14:27:52.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f937159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:52.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f937159a640 1 -- 172.21.15.26:0/3381778075 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9368001c00 con 0x7f93740b1730 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f9378825640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f9378825640 4 set_mon_vals no callback set 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f937159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f937159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.475+0000 7f937159a640 10 monclient: dump: 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.480 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.481 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.481 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:52.481 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:52.481 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:52.481 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:52.481 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 10 monclient: shutdown 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 1 -- 172.21.15.26:0/3381778075 >> v1:172.21.15.26:6789/0 conn(0x7f93740b1730 legacy=0x7f93740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 1 -- 172.21.15.26:0/3381778075 shutdown_connections 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 1 -- 172.21.15.26:0/3381778075 >> 172.21.15.26:0/3381778075 conn(0x7f93740617a0 msgr2=0x7f9374061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 1 -- 172.21.15.26:0/3381778075 shutdown_connections 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.476+0000 7f9378825640 1 -- 172.21.15.26:0/3381778075 wait complete. 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.477+0000 7f9378825640 10 monclient: build_initial_monmap 2024-03-25T14:27:52.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.477+0000 7f9378825640 10 monclient: monmap: 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.477+0000 7f9378825640 1 librados: starting msgr at 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.477+0000 7f9378825640 1 librados: starting objecter 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 Processor -- start 2024-03-25T14:27:52.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 -- start start 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 librados: setting wanted keys 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 librados: calling monclient init 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient: init 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient(hunting): picked mon.noname-c con 0x7f93740b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient(hunting): picked mon.noname-b con 0x7f93740b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient(hunting): picked mon.noname-a con 0x7f937414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374140930 con 0x7f937414be90 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374140380 con 0x7f93740b4d60 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374140510 con 0x7f93740b1730 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:52.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.478+0000 7f9378825640 10 monclient(hunting): authenticate will time out at 1557.581532s 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.488+0000 7f9371d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f93740b4d60 0x7f93741d7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60944/0 (socket says 172.21.15.26:60944) 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.488+0000 7f9371d9b640 1 -- 172.21.15.26:0/3475475542 learned_addr learned my addr 172.21.15.26:0/3475475542 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:27:52.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.489+0000 7f9362ffd640 10 client.?.objecter ms_handle_connect 0x7f93740b4d60 2024-03-25T14:27:52.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.489+0000 7f9362ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:52.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.496+0000 7f9362ffd640 10 client.?.objecter ms_handle_connect 0x7f937414be90 2024-03-25T14:27:52.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.496+0000 7f9362ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:52.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.497+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041871066 0 0) 0x7f9374140380 con 0x7f93740b4d60 2024-03-25T14:27:52.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.497+0000 7f9362ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.497+0000 7f9362ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.497+0000 7f9362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.497+0000 7f9362ffd640 10 monclient(hunting): my global_id is 4252 2024-03-25T14:27:52.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.497+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9354004eb0 con 0x7f93740b4d60 2024-03-25T14:27:52.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.504+0000 7f9362ffd640 10 client.?.objecter ms_handle_connect 0x7f93740b1730 2024-03-25T14:27:52.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.504+0000 7f9362ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:52.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.507+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960538045 0 0) 0x7f9374140930 con 0x7f937414be90 2024-03-25T14:27:52.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.507+0000 7f9362ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.507+0000 7f9362ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.507+0000 7f9362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.507+0000 7f9362ffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T14:27:52.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.507+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9374140380 con 0x7f937414be90 2024-03-25T14:27:52.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117315524 0 0) 0x7f9354004eb0 con 0x7f93740b4d60 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9374140930 con 0x7f93740b4d60 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9368001bc0 con 0x7f93740b4d60 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 10 monclient(hunting): dump: 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.517 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:52.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.511+0000 7f9362ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:52.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.520+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211828622 0 0) 0x7f9374140510 con 0x7f93740b1730 2024-03-25T14:27:52.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.520+0000 7f9362ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.520+0000 7f9362ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.520+0000 7f9362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.520+0000 7f9362ffd640 10 monclient(hunting): my global_id is 4250 2024-03-25T14:27:52.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.520+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9354004eb0 con 0x7f93740b1730 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 544947915 0 0) 0x7f9374140930 con 0x7f93740b4d60 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 >> v1:172.21.15.26:6790/0 conn(0x7f93740b1730 legacy=0x7f93741d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 >> v1:172.21.15.26:6789/0 conn(0x7f937414be90 legacy=0x7f93741e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 1 monclient: found mon.b 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9374144bc0 con 0x7f93740b4d60 2024-03-25T14:27:52.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 10 monclient: _finish_auth 0 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 5 monclient: authenticate success, global_id 4252 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 10 monclient: _renew_subs 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9374144db0 con 0x7f93740b4d60 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 10 monclient: _renew_subs 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:52.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9374137b80 con 0x7f93740b4d60 2024-03-25T14:27:52.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.526+0000 7f9378825640 1 librados: init done 2024-03-25T14:27:52.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.527+0000 7f9378825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:52.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.527+0000 7f9378825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:52.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.527+0000 7f9378825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:52.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.527+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93741401f0 con 0x7f93740b4d60 2024-03-25T14:27:52.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.809+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93680011f0 con 0x7f93740b4d60 2024-03-25T14:27:52.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.809+0000 7f9362ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:52.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.809+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9368003b00 con 0x7f93740b4d60 2024-03-25T14:27:52.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.809+0000 7f9362ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.810+0000 7f9362ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.810+0000 7f9362ffd640 10 monclient: dump: 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:27:52.815 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:27:52.816 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:27:52.816 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.809+0000 7f9370d99640 4 set_mon_vals no callback set 2024-03-25T14:27:52.816 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.810+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f9368003d20 con 0x7f93740b4d60 2024-03-25T14:27:52.816 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.811+0000 7f937259c640 10 monclient: get_auth_request con 0x7f935406ddb0 auth_method 0 2024-03-25T14:27:52.816 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.811+0000 7f9362ffd640 10 client.4252.objecter ms_handle_connect 0x7f935406ddb0 2024-03-25T14:27:52.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.815+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f9368054950 con 0x7f93740b4d60 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.815+0000 7f9362ffd640 10 client.4252.objecter ms_dispatch 0x7f9374000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.815+0000 7f9362ffd640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.815+0000 7f9362ffd640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.815+0000 7f9362ffd640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.816+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9368050290 con 0x7f93740b4d60 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.816+0000 7f9362ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:52.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:52.816+0000 7f9362ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:53.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.037+0000 7f9378825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:53.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.037+0000 7f9378825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:53.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.037+0000 7f9378825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:27:53.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.037+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9374140ae0 con 0x7f93740b4d60 2024-03-25T14:27:53.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.070+0000 7f9362ffd640 1 -- 172.21.15.26:0/3475475542 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f9374140ae0 con 0x7f93740b4d60 2024-03-25T14:27:53.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.070+0000 7f9362ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:53.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.070+0000 7f9362ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:53.075 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:27:53.075 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:53.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 10 librados: watch_flush enter 2024-03-25T14:27:53.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 10 librados: watch_flush exit 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 10 client.4252.objecter successfully canceled tick 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 >> v1:172.21.15.121:6800/288813066 conn(0x7f935406ddb0 legacy=0x7f9354070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 10 monclient: shutdown 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 >> v1:172.21.15.121:6789/0 conn(0x7f93740b4d60 legacy=0x7f93741d7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 shutdown_connections 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 >> 172.21.15.26:0/3475475542 conn(0x7f93740617a0 msgr2=0x7f93740b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 shutdown_connections 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 -- 172.21.15.26:0/3475475542 wait complete. 2024-03-25T14:27:53.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:27:53.073+0000 7f9378825640 1 librados: shutdown 2024-03-25T14:27:53.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:55.547 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:14.408 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.609 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:14.470 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.635 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:14.496 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.696 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:14.573 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.702 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:14.563 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.756 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:14.633 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.868 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:14.730 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.948 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:14.810 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:55.974 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:14.851 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:59.896 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:00.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.006+0000 7f0098def640 10 monclient: get_monmap_and_config 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.006+0000 7f0098def640 10 monclient: build_initial_monmap 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.006+0000 7f0098def640 10 monclient: monmap: 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:00.007418+0000 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:00.007418+0000 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:00.011 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.007+0000 7f0098def640 1 Processor -- start 2024-03-25T14:28:00.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.007+0000 7f0098def640 1 -- start start 2024-03-25T14:28:00.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.007+0000 7f0098def640 10 monclient: init 2024-03-25T14:28:00.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient(hunting): picked mon.noname-a con 0x7f00940a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient(hunting): picked mon.noname-b con 0x7f009413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient(hunting): picked mon.noname-c con 0x7f00941430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094142520 con 0x7f00940a98f0 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00940b28a0 con 0x7f00941430c0 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00940b2a30 con 0x7f009413f710 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:00.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.008+0000 7f0098def640 10 monclient(hunting): authenticate will time out at 1565.111020s 2024-03-25T14:28:00.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.016+0000 7f009259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f00940a98f0 0x7f00940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54026/0 (socket says 172.21.15.26:54026) 2024-03-25T14:28:00.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.016+0000 7f009259c640 1 -- 172.21.15.26:0/3936302618 learned_addr learned my addr 172.21.15.26:0/3936302618 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:00.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.024+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622130089 0 0) 0x7f0094142520 con 0x7f00940a98f0 2024-03-25T14:28:00.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.024+0000 7f0091d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.024+0000 7f0091d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.024+0000 7f0091d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.024+0000 7f0091d9b640 10 monclient(hunting): my global_id is 4272 2024-03-25T14:28:00.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.024+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0080003520 con 0x7f00940a98f0 2024-03-25T14:28:00.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.043+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001273801 0 0) 0x7f00940b28a0 con 0x7f00941430c0 2024-03-25T14:28:00.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.044+0000 7f0091d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.044+0000 7f0091d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.044+0000 7f0091d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.044+0000 7f0091d9b640 10 monclient(hunting): my global_id is 4255 2024-03-25T14:28:00.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.044+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0094142520 con 0x7f00941430c0 2024-03-25T14:28:00.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.045+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225873955 0 0) 0x7f00940b2a30 con 0x7f009413f710 2024-03-25T14:28:00.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.045+0000 7f0091d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.045+0000 7f0091d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.045+0000 7f0091d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.045+0000 7f0091d9b640 10 monclient(hunting): my global_id is 4253 2024-03-25T14:28:00.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.045+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00940b28a0 con 0x7f009413f710 2024-03-25T14:28:00.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433096074 0 0) 0x7f0080003520 con 0x7f00940a98f0 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 >> v1:172.21.15.26:6790/0 conn(0x7f009413f710 legacy=0x7f00941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 >> v1:172.21.15.121:6789/0 conn(0x7f00941430c0 legacy=0x7f009414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: _finish_hunting 0 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 1 monclient: found mon.noname-a 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940b44c0 con 0x7f00940a98f0 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: _finish_auth 0 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: _check_auth_tickets 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0084001a10 con 0x7f00940a98f0 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0098def640 5 monclient: authenticate success, global_id 4272 2024-03-25T14:28:00.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0098def640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0091d9b640 10 monclient: dump: 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.050+0000 7f0098def640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0084001b70 con 0x7f00940a98f0 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0091d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0091d9b640 1 -- 172.21.15.26:0/3936302618 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0084001f80 con 0x7f00940a98f0 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0091d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0091d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0091d9b640 10 monclient: dump: 2024-03-25T14:28:00.075 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0098def640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.071+0000 7f0098def640 4 set_mon_vals no callback set 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 10 monclient: shutdown 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 1 -- 172.21.15.26:0/3936302618 >> v1:172.21.15.26:6789/0 conn(0x7f00940a98f0 legacy=0x7f00940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 1 -- 172.21.15.26:0/3936302618 shutdown_connections 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 1 -- 172.21.15.26:0/3936302618 >> 172.21.15.26:0/3936302618 conn(0x7f00940b7ea0 msgr2=0x7f00940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 1 -- 172.21.15.26:0/3936302618 shutdown_connections 2024-03-25T14:28:00.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.072+0000 7f0098def640 1 -- 172.21.15.26:0/3936302618 wait complete. 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 10 monclient: build_initial_monmap 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 10 monclient: monmap: 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.080 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 1 librados: starting msgr at 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 1 librados: starting objecter 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 1 Processor -- start 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 1 -- start start 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 1 librados: setting wanted keys 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 1 librados: calling monclient init 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.073+0000 7f0098def640 10 monclient: init 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient(hunting): picked mon.noname-b con 0x7f00940a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient(hunting): picked mon.noname-c con 0x7f009413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient(hunting): picked mon.noname-a con 0x7f00941430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00941db2f0 con 0x7f00941430c0 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094139180 con 0x7f00940a98f0 2024-03-25T14:28:00.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094137670 con 0x7f009413f710 2024-03-25T14:28:00.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:00.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.074+0000 7f0098def640 10 monclient(hunting): authenticate will time out at 1565.177370s 2024-03-25T14:28:00.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.088+0000 7f009259c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f00940a98f0 0x7f00941dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:37684/0 (socket says 172.21.15.26:37684) 2024-03-25T14:28:00.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.088+0000 7f009259c640 1 -- 172.21.15.26:0/4050584133 learned_addr learned my addr 172.21.15.26:0/4050584133 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:00.095 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.091+0000 7f008b7fe640 10 client.?.objecter ms_handle_connect 0x7f00940a98f0 2024-03-25T14:28:00.095 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.091+0000 7f008b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:00.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.097+0000 7f008b7fe640 10 client.?.objecter ms_handle_connect 0x7f00941430c0 2024-03-25T14:28:00.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.097+0000 7f008b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:00.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.104+0000 7f008b7fe640 10 client.?.objecter ms_handle_connect 0x7f009413f710 2024-03-25T14:28:00.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.104+0000 7f008b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:00.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.106+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034258358 0 0) 0x7f0094139180 con 0x7f00940a98f0 2024-03-25T14:28:00.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.107+0000 7f008b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.107+0000 7f008b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.107+0000 7f008b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.107+0000 7f008b7fe640 10 monclient(hunting): my global_id is 4258 2024-03-25T14:28:00.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.107+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0074004eb0 con 0x7f00940a98f0 2024-03-25T14:28:00.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.111+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239820780 0 0) 0x7f00941db2f0 con 0x7f00941430c0 2024-03-25T14:28:00.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.112+0000 7f008b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.112+0000 7f008b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.112+0000 7f008b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.112+0000 7f008b7fe640 10 monclient(hunting): my global_id is 4275 2024-03-25T14:28:00.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.112+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0094139180 con 0x7f00941430c0 2024-03-25T14:28:00.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184543764 0 0) 0x7f0074004eb0 con 0x7f00940a98f0 2024-03-25T14:28:00.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00941db2f0 con 0x7f00940a98f0 2024-03-25T14:28:00.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0084001f40 con 0x7f00940a98f0 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 10 monclient(hunting): dump: 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:00.126 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.121+0000 7f008b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:00.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.130+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471857624 0 0) 0x7f0094137670 con 0x7f009413f710 2024-03-25T14:28:00.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.130+0000 7f008b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.130+0000 7f008b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.130+0000 7f008b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.130+0000 7f008b7fe640 10 monclient(hunting): my global_id is 4256 2024-03-25T14:28:00.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.130+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0074004eb0 con 0x7f009413f710 2024-03-25T14:28:00.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.134+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 757208905 0 0) 0x7f0094139180 con 0x7f00941430c0 2024-03-25T14:28:00.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.134+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0094137670 con 0x7f00941430c0 2024-03-25T14:28:00.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.134+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f008c0017d0 con 0x7f00941430c0 2024-03-25T14:28:00.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.134+0000 7f008b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.134+0000 7f008b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:00.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.134+0000 7f008b7fe640 10 monclient(hunting): dump: 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:00.139 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1005035284 0 0) 0x7f00941db2f0 con 0x7f00940a98f0 2024-03-25T14:28:00.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 >> v1:172.21.15.26:6790/0 conn(0x7f009413f710 legacy=0x7f0094132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 >> v1:172.21.15.26:6789/0 conn(0x7f00941430c0 legacy=0x7f00941e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:28:00.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 1 monclient: found mon.b 2024-03-25T14:28:00.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094137200 con 0x7f00940a98f0 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f008b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 5 monclient: authenticate success, global_id 4258 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 10 monclient: _renew_subs 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00940b7290 con 0x7f00940a98f0 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 10 monclient: _renew_subs 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:00.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0094133680 con 0x7f00940a98f0 2024-03-25T14:28:00.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.147+0000 7f0098def640 1 librados: init done 2024-03-25T14:28:00.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.148+0000 7f0098def640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.148+0000 7f0098def640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.148+0000 7f0098def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:00.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.148+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00940627b0 con 0x7f00940a98f0 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0084001570 con 0x7f00940a98f0 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f008b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0084005030 con 0x7f00940a98f0 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f008b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f008b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f008b7fe640 10 monclient: dump: 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.450 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:00.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.445+0000 7f009159a640 4 set_mon_vals no callback set 2024-03-25T14:28:00.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.446+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f008401b080 con 0x7f00940a98f0 2024-03-25T14:28:00.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.447+0000 7f0089d9b640 10 monclient: get_auth_request con 0x7f007406df20 auth_method 0 2024-03-25T14:28:00.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.447+0000 7f008b7fe640 10 client.4258.objecter ms_handle_connect 0x7f007406df20 2024-03-25T14:28:00.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f0084054d70 con 0x7f00940a98f0 2024-03-25T14:28:00.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 10 client.4258.objecter ms_dispatch 0x7f0094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:00.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:00.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:00.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T14:28:00.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00840506b0 con 0x7f00940a98f0 2024-03-25T14:28:00.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.452+0000 7f008b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:00.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.702+0000 7f0098def640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:00.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.702+0000 7f0098def640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:00.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.702+0000 7f0098def640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:00.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.702+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f009413e2b0 con 0x7f00940a98f0 2024-03-25T14:28:00.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.836+0000 7f008b7fe640 1 -- 172.21.15.26:0/4050584133 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f00840527b0 con 0x7f00940a98f0 2024-03-25T14:28:00.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.836+0000 7f008b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:00.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.836+0000 7f008b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:00.841 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:00.841 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:00.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 10 librados: watch_flush enter 2024-03-25T14:28:00.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 10 librados: watch_flush exit 2024-03-25T14:28:00.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T14:28:00.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 10 client.4258.objecter successfully canceled tick 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 >> v1:172.21.15.121:6800/288813066 conn(0x7f007406df20 legacy=0x7f00740703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 10 monclient: shutdown 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 >> v1:172.21.15.121:6789/0 conn(0x7f00940a98f0 legacy=0x7f00941dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 shutdown_connections 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.838+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 >> 172.21.15.26:0/4050584133 conn(0x7f00940b7ea0 msgr2=0x7f0094074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.839+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 shutdown_connections 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.839+0000 7f0098def640 1 -- 172.21.15.26:0/4050584133 wait complete. 2024-03-25T14:28:00.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:00.839+0000 7f0098def640 1 librados: shutdown 2024-03-25T14:28:00.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:07.662 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:07.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.776+0000 7fcb44c50640 10 monclient: get_monmap_and_config 2024-03-25T14:28:07.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.776+0000 7fcb44c50640 10 monclient: build_initial_monmap 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.776+0000 7fcb44c50640 10 monclient: monmap: 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:07.776895+0000 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:07.776895+0000 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:07.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.776+0000 7fcb44c50640 1 Processor -- start 2024-03-25T14:28:07.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.776+0000 7fcb44c50640 1 -- start start 2024-03-25T14:28:07.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.776+0000 7fcb44c50640 10 monclient: init 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient(hunting): picked mon.noname-c con 0x7fcb400a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient(hunting): picked mon.noname-b con 0x7fcb4013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient(hunting): picked mon.noname-a con 0x7fcb401430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb40142520 con 0x7fcb401430c0 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb400b28a0 con 0x7fcb400a98f0 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb400b2a30 con 0x7fcb4013f710 2024-03-25T14:28:07.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:07.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.777+0000 7fcb44c50640 10 monclient(hunting): authenticate will time out at 1572.880462s 2024-03-25T14:28:07.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.787+0000 7fcb3e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcb400a98f0 0x7fcb400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:52198/0 (socket says 172.21.15.26:52198) 2024-03-25T14:28:07.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.787+0000 7fcb3e59c640 1 -- 172.21.15.26:0/1318864411 learned_addr learned my addr 172.21.15.26:0/1318864411 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:07.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.792+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922508565 0 0) 0x7fcb400b28a0 con 0x7fcb400a98f0 2024-03-25T14:28:07.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.792+0000 7fcb3d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.792+0000 7fcb3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.792+0000 7fcb3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.792+0000 7fcb3d59a640 10 monclient(hunting): my global_id is 4261 2024-03-25T14:28:07.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.792+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb28003520 con 0x7fcb400a98f0 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.808+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2123221094 0 0) 0x7fcb28003520 con 0x7fcb400a98f0 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.808+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 >> v1:172.21.15.26:6790/0 conn(0x7fcb4013f710 legacy=0x7fcb401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 >> v1:172.21.15.26:6789/0 conn(0x7fcb401430c0 legacy=0x7fcb4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: _finish_hunting 0 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 1 monclient: found mon.noname-c 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb400b44c0 con 0x7fcb400a98f0 2024-03-25T14:28:07.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: _finish_auth 0 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: _check_auth_tickets 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fcb30001a10 con 0x7fcb400a98f0 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb44c50640 5 monclient: authenticate success, global_id 4261 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb44c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb3d59a640 10 monclient: dump: 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:07.814 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:07.815 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:07.815 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:07.815 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:07.815 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:07.815 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:07.815 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.809+0000 7fcb44c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:07.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb30001b70 con 0x7fcb400a98f0 2024-03-25T14:28:07.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:07.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb3d59a640 1 -- 172.21.15.26:0/1318864411 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fcb30001f80 con 0x7fcb400a98f0 2024-03-25T14:28:07.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb3d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb3d59a640 10 monclient: dump: 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:07.826 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:07.827 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:07.827 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:07.827 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:07.827 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:07.827 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb44c50640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:07.827 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.821+0000 7fcb44c50640 4 set_mon_vals no callback set 2024-03-25T14:28:07.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 10 monclient: shutdown 2024-03-25T14:28:07.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:07.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 1 -- 172.21.15.26:0/1318864411 >> v1:172.21.15.121:6789/0 conn(0x7fcb400a98f0 legacy=0x7fcb400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 1 -- 172.21.15.26:0/1318864411 shutdown_connections 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 1 -- 172.21.15.26:0/1318864411 >> 172.21.15.26:0/1318864411 conn(0x7fcb400b7ea0 msgr2=0x7fcb400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 1 -- 172.21.15.26:0/1318864411 shutdown_connections 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.822+0000 7fcb44c50640 1 -- 172.21.15.26:0/1318864411 wait complete. 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 10 monclient: build_initial_monmap 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 10 monclient: monmap: 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:07.830 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 1 librados: starting msgr at 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 1 librados: starting objecter 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 1 Processor -- start 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 1 -- start start 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 1 librados: setting wanted keys 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 1 librados: calling monclient init 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.823+0000 7fcb44c50640 10 monclient: init 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient(hunting): picked mon.noname-c con 0x7fcb400a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient(hunting): picked mon.noname-b con 0x7fcb4013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:07.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient(hunting): picked mon.noname-a con 0x7fcb401430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:07.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb40138b20 con 0x7fcb401430c0 2024-03-25T14:28:07.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb40137a80 con 0x7fcb4013f710 2024-03-25T14:28:07.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb40139880 con 0x7fcb400a98f0 2024-03-25T14:28:07.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:07.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.824+0000 7fcb44c50640 10 monclient(hunting): authenticate will time out at 1572.926986s 2024-03-25T14:28:07.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.831+0000 7fcb3dd9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcb4013f710 0x7fcb40139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:52240/0 (socket says 172.21.15.26:52240) 2024-03-25T14:28:07.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.831+0000 7fcb3dd9b640 1 -- 172.21.15.26:0/3663306497 learned_addr learned my addr 172.21.15.26:0/3663306497 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:07.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.835+0000 7fcb2effd640 10 client.?.objecter ms_handle_connect 0x7fcb4013f710 2024-03-25T14:28:07.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.835+0000 7fcb2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:07.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.841+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443501301 0 0) 0x7fcb40137a80 con 0x7fcb4013f710 2024-03-25T14:28:07.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.841+0000 7fcb2effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.841+0000 7fcb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.841+0000 7fcb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.841+0000 7fcb2effd640 10 monclient(hunting): my global_id is 4264 2024-03-25T14:28:07.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.841+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb20004eb0 con 0x7fcb4013f710 2024-03-25T14:28:07.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.845+0000 7fcb2effd640 10 client.?.objecter ms_handle_connect 0x7fcb401430c0 2024-03-25T14:28:07.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.845+0000 7fcb2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:07.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.847+0000 7fcb2effd640 10 client.?.objecter ms_handle_connect 0x7fcb400a98f0 2024-03-25T14:28:07.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.847+0000 7fcb2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812622725 0 0) 0x7fcb20004eb0 con 0x7fcb4013f710 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb40137a80 con 0x7fcb4013f710 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fcb34001a30 con 0x7fcb4013f710 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 10 monclient(hunting): dump: 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:07.863 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:07.864 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:07.864 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:07.864 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:07.864 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:07.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.858+0000 7fcb2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:07.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.867+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703141252 0 0) 0x7fcb40138b20 con 0x7fcb401430c0 2024-03-25T14:28:07.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.867+0000 7fcb2effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.867+0000 7fcb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.867+0000 7fcb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.867+0000 7fcb2effd640 10 monclient(hunting): my global_id is 4281 2024-03-25T14:28:07.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.867+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb20004eb0 con 0x7fcb401430c0 2024-03-25T14:28:07.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.872+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724995888 0 0) 0x7fcb40139880 con 0x7fcb400a98f0 2024-03-25T14:28:07.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.872+0000 7fcb2effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.872+0000 7fcb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.872+0000 7fcb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.872+0000 7fcb2effd640 10 monclient(hunting): my global_id is 4262 2024-03-25T14:28:07.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.872+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb40138b20 con 0x7fcb400a98f0 2024-03-25T14:28:07.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2861977141 0 0) 0x7fcb40137a80 con 0x7fcb4013f710 2024-03-25T14:28:07.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 >> v1:172.21.15.26:6790/0 conn(0x7fcb400a98f0 legacy=0x7fcb40137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 >> v1:172.21.15.26:6789/0 conn(0x7fcb401430c0 legacy=0x7fcb401e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 1 monclient: found mon.b 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40139a10 con 0x7fcb4013f710 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 10 monclient: _finish_auth 0 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 5 monclient: authenticate success, global_id 4264 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 10 monclient: _renew_subs 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb401dab00 con 0x7fcb4013f710 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 10 monclient: _renew_subs 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4013b4f0 con 0x7fcb4013f710 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.874+0000 7fcb44c50640 1 librados: init done 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.875+0000 7fcb44c50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.875+0000 7fcb44c50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.875+0000 7fcb44c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:07.875+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb400627b0 con 0x7fcb4013f710 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb34001cb0 con 0x7fcb4013f710 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fcb34003a60 con 0x7fcb4013f710 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb3cd99640 4 set_mon_vals no callback set 2024-03-25T14:28:08.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.159+0000 7fcb2effd640 10 monclient: dump: 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.160+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fcb34003c40 con 0x7fcb4013f710 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.161+0000 7fcb3e59c640 10 monclient: get_auth_request con 0x7fcb2006de60 auth_method 0 2024-03-25T14:28:08.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.161+0000 7fcb2effd640 10 client.4264.objecter ms_handle_connect 0x7fcb2006de60 2024-03-25T14:28:08.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fcb34054860 con 0x7fcb4013f710 2024-03-25T14:28:08.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 10 client.4264.objecter ms_dispatch 0x7fcb40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:08.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:08.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:08.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T14:28:08.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb34050370 con 0x7fcb4013f710 2024-03-25T14:28:08.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:08.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.165+0000 7fcb2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:08.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.424+0000 7fcb44c50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:08.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.424+0000 7fcb44c50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:08.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.424+0000 7fcb44c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:08.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.424+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb400b3ed0 con 0x7fcb4013f710 2024-03-25T14:28:08.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.450+0000 7fcb2effd640 1 -- 172.21.15.26:0/3663306497 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fcb340522a0 con 0x7fcb4013f710 2024-03-25T14:28:08.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.450+0000 7fcb2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:08.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.450+0000 7fcb2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:08.454 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:08.454 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 10 librados: watch_flush enter 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 10 librados: watch_flush exit 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 10 client.4264.objecter successfully canceled tick 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 >> v1:172.21.15.121:6800/288813066 conn(0x7fcb2006de60 legacy=0x7fcb20070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 10 monclient: shutdown 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 >> v1:172.21.15.121:6789/0 conn(0x7fcb4013f710 legacy=0x7fcb40139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:08.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 shutdown_connections 2024-03-25T14:28:08.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.452+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 >> 172.21.15.26:0/3663306497 conn(0x7fcb400b7ea0 msgr2=0x7fcb40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:08.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.453+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 shutdown_connections 2024-03-25T14:28:08.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.453+0000 7fcb44c50640 1 -- 172.21.15.26:0/3663306497 wait complete. 2024-03-25T14:28:08.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:08.453+0000 7fcb44c50640 1 librados: shutdown 2024-03-25T14:28:08.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:15.275 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:15.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.386+0000 7f4c709e6640 10 monclient: get_monmap_and_config 2024-03-25T14:28:15.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.386+0000 7f4c709e6640 10 monclient: build_initial_monmap 2024-03-25T14:28:15.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.386+0000 7f4c709e6640 10 monclient: monmap: 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:15.388212+0000 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:15.388212+0000 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:15.392 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.387+0000 7f4c709e6640 1 Processor -- start 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.387+0000 7f4c709e6640 1 -- start start 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.387+0000 7f4c709e6640 10 monclient: init 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6c13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6c1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:15.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c142520 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c0b28a0 con 0x7f4c6c13f710 2024-03-25T14:28:15.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c0b2a30 con 0x7f4c6c1430c0 2024-03-25T14:28:15.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:15.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.388+0000 7f4c709e6640 10 monclient(hunting): authenticate will time out at 1580.492253s 2024-03-25T14:28:15.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.396+0000 7f4c6ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4c6c1430c0 0x7f4c6c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50184/0 (socket says 172.21.15.26:50184) 2024-03-25T14:28:15.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.396+0000 7f4c6ad9d640 1 -- 172.21.15.26:0/3222471812 learned_addr learned my addr 172.21.15.26:0/3222471812 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:15.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.408+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966937196 0 0) 0x7f4c6c0b28a0 con 0x7f4c6c13f710 2024-03-25T14:28:15.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.408+0000 7f4c6959a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.408+0000 7f4c6959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.408+0000 7f4c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.408+0000 7f4c6959a640 10 monclient(hunting): my global_id is 4267 2024-03-25T14:28:15.414 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.408+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c54003520 con 0x7f4c6c13f710 2024-03-25T14:28:15.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.414+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54922048 0 0) 0x7f4c6c0b2a30 con 0x7f4c6c1430c0 2024-03-25T14:28:15.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.414+0000 7f4c6959a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.414+0000 7f4c6959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.414+0000 7f4c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.414+0000 7f4c6959a640 10 monclient(hunting): my global_id is 4265 2024-03-25T14:28:15.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.414+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6c0b28a0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 896361259 0 0) 0x7f4c6c0b28a0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 >> v1:172.21.15.121:6789/0 conn(0x7f4c6c13f710 legacy=0x7f4c6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a98f0 legacy=0x7f4c6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: _finish_hunting 0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 1 monclient: found mon.noname-b 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c0b44c0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: _finish_auth 0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: _check_auth_tickets 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c640019f0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c6959a640 10 monclient: dump: 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c709e6640 5 monclient: authenticate success, global_id 4265 2024-03-25T14:28:15.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.423+0000 7f4c709e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:15.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c64001b50 con 0x7f4c6c1430c0 2024-03-25T14:28:15.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:15.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c6959a640 1 -- 172.21.15.26:0/3222471812 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c64001f20 con 0x7f4c6c1430c0 2024-03-25T14:28:15.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c6959a640 10 monclient: dump: 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c709e6640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:15.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.427+0000 7f4c709e6640 4 set_mon_vals no callback set 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 10 monclient: shutdown 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 1 -- 172.21.15.26:0/3222471812 >> v1:172.21.15.26:6790/0 conn(0x7f4c6c1430c0 legacy=0x7f4c6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 1 -- 172.21.15.26:0/3222471812 shutdown_connections 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 1 -- 172.21.15.26:0/3222471812 >> 172.21.15.26:0/3222471812 conn(0x7f4c6c0b7ea0 msgr2=0x7f4c6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 1 -- 172.21.15.26:0/3222471812 shutdown_connections 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.428+0000 7f4c709e6640 1 -- 172.21.15.26:0/3222471812 wait complete. 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.429+0000 7f4c709e6640 10 monclient: build_initial_monmap 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.429+0000 7f4c709e6640 10 monclient: monmap: 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:15.435 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.429+0000 7f4c709e6640 1 librados: starting msgr at 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.429+0000 7f4c709e6640 1 librados: starting objecter 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.429+0000 7f4c709e6640 1 Processor -- start 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 1 -- start start 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 1 librados: setting wanted keys 2024-03-25T14:28:15.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 1 librados: calling monclient init 2024-03-25T14:28:15.437 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient: init 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c137e20 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c0b1720 con 0x7f4c6c1430c0 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c138710 con 0x7f4c6c13f710 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:15.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.430+0000 7f4c709e6640 10 monclient(hunting): authenticate will time out at 1580.534443s 2024-03-25T14:28:15.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.439+0000 7f4c6ad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4c6c1430c0 0x7f4c6c1daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:56968/0 (socket says 172.21.15.26:56968) 2024-03-25T14:28:15.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.439+0000 7f4c6ad9d640 1 -- 172.21.15.26:0/460891492 learned_addr learned my addr 172.21.15.26:0/460891492 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:15.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.440+0000 7f4c5affd640 10 client.?.objecter ms_handle_connect 0x7f4c6c1430c0 2024-03-25T14:28:15.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.440+0000 7f4c5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:15.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.441+0000 7f4c5affd640 10 client.?.objecter ms_handle_connect 0x7f4c6c0a98f0 2024-03-25T14:28:15.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.441+0000 7f4c5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:15.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.452+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617255679 0 0) 0x7f4c6c0b1720 con 0x7f4c6c1430c0 2024-03-25T14:28:15.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.452+0000 7f4c5affd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.452+0000 7f4c5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.452+0000 7f4c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.452+0000 7f4c5affd640 10 monclient(hunting): my global_id is 4270 2024-03-25T14:28:15.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.452+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c4c004eb0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.461+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871453609 0 0) 0x7f4c6c137e20 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.461+0000 7f4c5affd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.461+0000 7f4c5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.461+0000 7f4c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.461+0000 7f4c5affd640 10 monclient(hunting): my global_id is 4287 2024-03-25T14:28:15.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.461+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6c0b1720 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.466+0000 7f4c5affd640 10 client.?.objecter ms_handle_connect 0x7f4c6c13f710 2024-03-25T14:28:15.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.466+0000 7f4c5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:15.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.471+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657161760 0 0) 0x7f4c4c004eb0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.471+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c6c137e20 con 0x7f4c6c1430c0 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.479+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c64001fc0 con 0x7f4c6c1430c0 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.479+0000 7f4c5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.479+0000 7f4c5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.479+0000 7f4c5affd640 10 monclient(hunting): dump: 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.484 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.485 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.479+0000 7f4c5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 121918882 0 0) 0x7f4c6c0b1720 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c4c004eb0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c5c0018c0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): dump: 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.491 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023585022 0 0) 0x7f4c6c138710 con 0x7f4c6c13f710 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 10 monclient(hunting): my global_id is 4268 2024-03-25T14:28:15.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.486+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6c0b1720 con 0x7f4c6c13f710 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2977085681 0 0) 0x7f4c4c004eb0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 >> v1:172.21.15.26:6790/0 conn(0x7f4c6c13f710 legacy=0x7f4c6c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 >> v1:172.21.15.121:6789/0 conn(0x7f4c6c1430c0 legacy=0x7f4c6c1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 1 monclient: found mon.a 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c138040 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 10 monclient: _finish_auth 0 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:15.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 5 monclient: authenticate success, global_id 4287 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 10 monclient: _renew_subs 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c1382f0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 10 monclient: _renew_subs 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c1e2fb0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.499+0000 7f4c709e6640 1 librados: init done 2024-03-25T14:28:15.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.500+0000 7f4c709e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:15.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.501+0000 7f4c709e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:15.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.501+0000 7f4c709e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:15.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.501+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c0b3ef0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.828 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c5c0035f0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.828 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:15.828 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c5c003a00 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c5affd640 10 monclient: dump: 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:15.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.823+0000 7f4c68d99640 4 set_mon_vals no callback set 2024-03-25T14:28:15.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.824+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4c5c003c20 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.824+0000 7f4c69d9b640 10 monclient: get_auth_request con 0x7f4c4c06def0 auth_method 0 2024-03-25T14:28:15.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.825+0000 7f4c5affd640 10 client.4287.objecter ms_handle_connect 0x7f4c4c06def0 2024-03-25T14:28:15.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f4c5c0535a0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 10 client.4287.objecter ms_dispatch 0x7f4c6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:15.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:15.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:15.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T14:28:15.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c5c04f0b0 con 0x7f4c6c0a98f0 2024-03-25T14:28:15.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:15.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:15.830+0000 7f4c5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:15.901 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:34.762 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:15.907 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:34.769 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:15.953 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:34.814 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:15.961 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:34.838 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:15.964 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:34.825 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:15.970 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:34.847 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:15.976 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:34.838 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:16.006 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:34.883 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:16.024 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:34.885 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:16.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.042+0000 7f4c709e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:16.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.042+0000 7f4c709e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:16.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.042+0000 7f4c709e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:16.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.042+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c6c0627b0 con 0x7f4c6c0a98f0 2024-03-25T14:28:16.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.130+0000 7f4c5affd640 1 -- 172.21.15.26:0/460891492 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f4c5c050fe0 con 0x7f4c6c0a98f0 2024-03-25T14:28:16.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.130+0000 7f4c5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:16.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.130+0000 7f4c5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:16.136 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:16.136 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.133+0000 7f4c709e6640 10 librados: watch_flush enter 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.133+0000 7f4c709e6640 10 librados: watch_flush exit 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.133+0000 7f4c709e6640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.133+0000 7f4c709e6640 10 client.4287.objecter successfully canceled tick 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.133+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 >> v1:172.21.15.121:6800/288813066 conn(0x7f4c4c06def0 legacy=0x7f4c4c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 10 monclient: shutdown 2024-03-25T14:28:16.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:16.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a98f0 legacy=0x7f4c6c137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:16.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 shutdown_connections 2024-03-25T14:28:16.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 >> 172.21.15.26:0/460891492 conn(0x7f4c6c0b7ea0 msgr2=0x7f4c6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:16.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 shutdown_connections 2024-03-25T14:28:16.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 1 -- 172.21.15.26:0/460891492 wait complete. 2024-03-25T14:28:16.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:16.134+0000 7f4c709e6640 1 librados: shutdown 2024-03-25T14:28:16.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:22.962 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:23.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.071+0000 7f9839eb3640 10 monclient: get_monmap_and_config 2024-03-25T14:28:23.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.071+0000 7f9839eb3640 10 monclient: build_initial_monmap 2024-03-25T14:28:23.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.071+0000 7f9839eb3640 10 monclient: monmap: 2024-03-25T14:28:23.076 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:23.073155+0000 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:23.073155+0000 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:23.077 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.072+0000 7f9839eb3640 1 Processor -- start 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.072+0000 7f9839eb3640 1 -- start start 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.072+0000 7f9839eb3640 10 monclient: init 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.072+0000 7f9839eb3640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.072+0000 7f9839eb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f98340b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f98340b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f983414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340af940 con 0x7f983414be90 2024-03-25T14:28:23.079 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340b0a70 con 0x7f98340b1730 2024-03-25T14:28:23.080 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340b0d50 con 0x7f98340b4d60 2024-03-25T14:28:23.080 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:23.080 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.073+0000 7f9839eb3640 10 monclient(hunting): authenticate will time out at 1588.176699s 2024-03-25T14:28:23.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.085+0000 7f9832ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f98340b1730 0x7f98340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:40128/0 (socket says 172.21.15.26:40128) 2024-03-25T14:28:23.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.085+0000 7f9832ffd640 1 -- 172.21.15.26:0/391679164 learned_addr learned my addr 172.21.15.26:0/391679164 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:23.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.094+0000 7f9833fff640 1 -- 172.21.15.26:0/391679164 >> v1:172.21.15.26:6789/0 conn(0x7f983414be90 legacy=0x7f983414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T14:28:23.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.094+0000 7f9833fff640 1 -- 172.21.15.26:0/391679164 >> v1:172.21.15.26:6789/0 conn(0x7f983414be90 legacy=0x7f983414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:28:23.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.094+0000 7f9833fff640 1 --1- 172.21.15.26:0/391679164 >> v1:172.21.15.26:6789/0 conn(0x7f983414be90 0x7f983414e280 :-1 s=OPENED pgs=5582 cs=1 l=1).handle_message read tag failed 2024-03-25T14:28:23.100 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.094+0000 7f9833fff640 1 --1- 172.21.15.26:0/391679164 >> v1:172.21.15.26:6789/0 conn(0x7f983414be90 0x7f983414e280 :-1 s=OPENED pgs=5582 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:28:23.100 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.094+0000 7f98327fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-03-25T14:28:23.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.100+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228500086 0 0) 0x7f98340b0a70 con 0x7f98340b1730 2024-03-25T14:28:23.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.100+0000 7f98327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.100+0000 7f98327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.100+0000 7f98327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.100+0000 7f98327fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T14:28:23.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.100+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f981c003520 con 0x7f98340b1730 2024-03-25T14:28:23.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.110+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323877248 0 0) 0x7f98340b0d50 con 0x7f98340b4d60 2024-03-25T14:28:23.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.110+0000 7f98327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.110+0000 7f98327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.110+0000 7f98327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.110+0000 7f98327fc640 10 monclient(hunting): my global_id is 4271 2024-03-25T14:28:23.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.110+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98340b0a70 con 0x7f98340b4d60 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623593280 0 0) 0x7f981c003520 con 0x7f98340b1730 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 >> v1:172.21.15.26:6790/0 conn(0x7f98340b4d60 legacy=0x7f983413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 >> v1:172.21.15.26:6789/0 conn(0x7f983414be90 legacy=0x7f983414e280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 10 monclient: _finish_hunting 0 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 1 monclient: found mon.noname-c 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:23.127 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340b1030 con 0x7f98340b1730 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 10 monclient: _finish_auth 0 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 10 monclient: _check_auth_tickets 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.121+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9828001690 con 0x7f98340b1730 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.122+0000 7f98327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.122+0000 7f98327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.122+0000 7f98327fc640 10 monclient: dump: 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.128 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.129 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:23.129 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:23.129 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:23.129 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.129 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.122+0000 7f9839eb3640 5 monclient: authenticate success, global_id 4273 2024-03-25T14:28:23.129 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.122+0000 7f9839eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.164+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98280017f0 con 0x7f98340b1730 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.164+0000 7f98327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.164+0000 7f98327fc640 1 -- 172.21.15.26:0/391679164 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9828001c00 con 0x7f98340b1730 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.164+0000 7f9839eb3640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.164+0000 7f9839eb3640 4 set_mon_vals no callback set 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.164+0000 7f98327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f98327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f98327fc640 10 monclient: dump: 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:23.170 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:23.171 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 10 monclient: shutdown 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 1 -- 172.21.15.26:0/391679164 >> v1:172.21.15.121:6789/0 conn(0x7f98340b1730 legacy=0x7f98340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 1 -- 172.21.15.26:0/391679164 shutdown_connections 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 1 -- 172.21.15.26:0/391679164 >> 172.21.15.26:0/391679164 conn(0x7f98340617a0 msgr2=0x7f9834061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 1 -- 172.21.15.26:0/391679164 shutdown_connections 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.165+0000 7f9839eb3640 1 -- 172.21.15.26:0/391679164 wait complete. 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.166+0000 7f9839eb3640 10 monclient: build_initial_monmap 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.166+0000 7f9839eb3640 10 monclient: monmap: 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:23.174 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.166+0000 7f9839eb3640 1 librados: starting msgr at 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.166+0000 7f9839eb3640 1 librados: starting objecter 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.166+0000 7f9839eb3640 1 Processor -- start 2024-03-25T14:28:23.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 1 -- start start 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 1 librados: setting wanted keys 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 1 librados: calling monclient init 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient: init 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f98340b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f98340b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f983414be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834144930 con 0x7f98340b1730 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834144c10 con 0x7f983414be90 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834143890 con 0x7f98340b4d60 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:23.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.167+0000 7f9839eb3640 10 monclient(hunting): authenticate will time out at 1588.271488s 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.178+0000 7f98337fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f98340b1730 0x7f9834145030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37540/0 (socket says 172.21.15.26:37540) 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.178+0000 7f98337fe640 1 -- 172.21.15.26:0/775848969 learned_addr learned my addr 172.21.15.26:0/775848969 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:23.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.181+0000 7f981bfff640 10 client.?.objecter ms_handle_connect 0x7f98340b1730 2024-03-25T14:28:23.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.181+0000 7f981bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:23.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.195+0000 7f981bfff640 10 client.?.objecter ms_handle_connect 0x7f98340b4d60 2024-03-25T14:28:23.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.195+0000 7f981bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.196+0000 7f981bfff640 10 client.?.objecter ms_handle_connect 0x7f983414be90 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.196+0000 7f981bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:23.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.200+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194223568 0 0) 0x7f9834144930 con 0x7f98340b1730 2024-03-25T14:28:23.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.200+0000 7f981bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.200+0000 7f981bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.200+0000 7f981bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.200+0000 7f981bfff640 10 monclient(hunting): my global_id is 4293 2024-03-25T14:28:23.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.200+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820004eb0 con 0x7f98340b1730 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.207+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510062541 0 0) 0x7f9834144c10 con 0x7f983414be90 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.207+0000 7f981bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.207+0000 7f981bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.207+0000 7f981bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.207+0000 7f981bfff640 10 monclient(hunting): my global_id is 4276 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.207+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9834144930 con 0x7f983414be90 2024-03-25T14:28:23.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.214+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900827312 0 0) 0x7f9834143890 con 0x7f98340b4d60 2024-03-25T14:28:23.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.214+0000 7f981bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.214+0000 7f981bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.214+0000 7f981bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.214+0000 7f981bfff640 10 monclient(hunting): my global_id is 4274 2024-03-25T14:28:23.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.214+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9834144c10 con 0x7f98340b4d60 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.230+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 860343990 0 0) 0x7f9820004eb0 con 0x7f98340b1730 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.230+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9834143890 con 0x7f98340b1730 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.230+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9824001a30 con 0x7f98340b1730 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.230+0000 7f981bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.230+0000 7f981bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.230+0000 7f981bfff640 10 monclient(hunting): dump: 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.236 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.231+0000 7f981bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:23.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.238+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292407737 0 0) 0x7f9834144930 con 0x7f983414be90 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.238+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9820004eb0 con 0x7f983414be90 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.238+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f982c001bb0 con 0x7f983414be90 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.238+0000 7f981bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.238+0000 7f981bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.238+0000 7f981bfff640 10 monclient(hunting): dump: 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.244 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:23.245 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:23.245 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:23.245 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.244+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545595761 0 0) 0x7f9834144c10 con 0x7f98340b4d60 2024-03-25T14:28:23.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.244+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9834144930 con 0x7f98340b4d60 2024-03-25T14:28:23.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.244+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9828001b00 con 0x7f98340b4d60 2024-03-25T14:28:23.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.244+0000 7f981bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.244+0000 7f981bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:23.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.244+0000 7f981bfff640 10 monclient(hunting): dump: 2024-03-25T14:28:23.250 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2435733266 0 0) 0x7f9820004eb0 con 0x7f983414be90 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 >> v1:172.21.15.26:6790/0 conn(0x7f98340b4d60 legacy=0x7f983413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 >> v1:172.21.15.26:6789/0 conn(0x7f98340b1730 legacy=0x7f9834145030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 1 monclient: found mon.b 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:23.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9834143b70 con 0x7f983414be90 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 10 monclient: _finish_auth 0 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f981bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 5 monclient: authenticate success, global_id 4276 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 10 monclient: _renew_subs 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9834145740 con 0x7f983414be90 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 10 monclient: _renew_subs 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f983413eea0 con 0x7f983414be90 2024-03-25T14:28:23.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.259+0000 7f9839eb3640 1 librados: init done 2024-03-25T14:28:23.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.260+0000 7f9839eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:23.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.260+0000 7f9839eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:23.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.260+0000 7f9839eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:23.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.260+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98341446e0 con 0x7f983414be90 2024-03-25T14:28:23.554 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.548+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f982c002060 con 0x7f983414be90 2024-03-25T14:28:23.554 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f981bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:23.554 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f982c003b10 con 0x7f983414be90 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f981bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f981bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f981bfff640 10 monclient: dump: 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:23.555 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:23.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f9831ffb640 4 set_mon_vals no callback set 2024-03-25T14:28:23.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.549+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f982c003d30 con 0x7f983414be90 2024-03-25T14:28:23.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.550+0000 7f98337fe640 10 monclient: get_auth_request con 0x7f982006e110 auth_method 0 2024-03-25T14:28:23.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.550+0000 7f981bfff640 10 client.4276.objecter ms_handle_connect 0x7f982006e110 2024-03-25T14:28:23.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.554+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f982c054930 con 0x7f983414be90 2024-03-25T14:28:23.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.554+0000 7f981bfff640 10 client.4276.objecter ms_dispatch 0x7f9834000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:23.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.554+0000 7f981bfff640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:23.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.554+0000 7f981bfff640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:23.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.554+0000 7f981bfff640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T14:28:23.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.555+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f982c050270 con 0x7f983414be90 2024-03-25T14:28:23.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.555+0000 7f981bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:23.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.555+0000 7f981bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:23.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.808+0000 7f9839eb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:23.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.808+0000 7f9839eb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:23.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.808+0000 7f9839eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:23.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.808+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f983413fc40 con 0x7f983414be90 2024-03-25T14:28:23.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.897+0000 7f981bfff640 1 -- 172.21.15.26:0/775848969 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f982c052370 con 0x7f983414be90 2024-03-25T14:28:23.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.897+0000 7f981bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:23.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.897+0000 7f981bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:23.903 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:23.903 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:23.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 10 librados: watch_flush enter 2024-03-25T14:28:23.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 10 librados: watch_flush exit 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 10 client.4276.objecter successfully canceled tick 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 >> v1:172.21.15.121:6800/288813066 conn(0x7f982006e110 legacy=0x7f98200705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 10 monclient: shutdown 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 >> v1:172.21.15.121:6789/0 conn(0x7f983414be90 legacy=0x7f983413d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 shutdown_connections 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 >> 172.21.15.26:0/775848969 conn(0x7f98340617a0 msgr2=0x7f983413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:23.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 shutdown_connections 2024-03-25T14:28:23.907 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 -- 172.21.15.26:0/775848969 wait complete. 2024-03-25T14:28:23.907 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:23.900+0000 7f9839eb3640 1 librados: shutdown 2024-03-25T14:28:23.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:30.725 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.840+0000 7f04a76ae640 10 monclient: get_monmap_and_config 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.840+0000 7f04a76ae640 10 monclient: build_initial_monmap 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.840+0000 7f04a76ae640 10 monclient: monmap: 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:30.840993+0000 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:30.840993+0000 2024-03-25T14:28:30.844 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:30.845 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:30.845 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:30.845 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:30.845 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:30.845 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.840+0000 7f04a76ae640 1 Processor -- start 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 1 -- start start 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 10 monclient: init 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 10 monclient(hunting): picked mon.noname-c con 0x7f04a00a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 10 monclient(hunting): picked mon.noname-b con 0x7f04a013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 10 monclient(hunting): picked mon.noname-a con 0x7f04a01430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a0142520 con 0x7f04a01430c0 2024-03-25T14:28:30.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00b28a0 con 0x7f04a00a98f0 2024-03-25T14:28:30.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.841+0000 7f04a76ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00b2a30 con 0x7f04a013f710 2024-03-25T14:28:30.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.842+0000 7f04a76ae640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:30.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.842+0000 7f04a76ae640 10 monclient(hunting): authenticate will time out at 1595.944604s 2024-03-25T14:28:30.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.859+0000 7f04a544a640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f04a00a98f0 0x7f04a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39326/0 (socket says 172.21.15.26:39326) 2024-03-25T14:28:30.863 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.859+0000 7f04a544a640 1 -- 172.21.15.26:0/1722443441 learned_addr learned my addr 172.21.15.26:0/1722443441 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:30.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.867+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85484568 0 0) 0x7f04a00b28a0 con 0x7f04a00a98f0 2024-03-25T14:28:30.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.867+0000 7f0497fff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:30.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.867+0000 7f0497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:30.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.867+0000 7f0497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:30.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.867+0000 7f0497fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T14:28:30.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.867+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003520 con 0x7f04a00a98f0 2024-03-25T14:28:30.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.874+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857695881 0 0) 0x7f04a00b2a30 con 0x7f04a013f710 2024-03-25T14:28:30.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.874+0000 7f0497fff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:30.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.874+0000 7f0497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:30.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.874+0000 7f0497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:30.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.874+0000 7f0497fff640 10 monclient(hunting): my global_id is 4277 2024-03-25T14:28:30.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.874+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a00b28a0 con 0x7f04a013f710 2024-03-25T14:28:30.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1708845460 0 0) 0x7f048c003520 con 0x7f04a00a98f0 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 >> v1:172.21.15.26:6790/0 conn(0x7f04a013f710 legacy=0x7f04a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 >> v1:172.21.15.26:6789/0 conn(0x7f04a01430c0 legacy=0x7f04a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: _finish_hunting 0 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 1 monclient: found mon.noname-c 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a00b44c0 con 0x7f04a00a98f0 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: _finish_auth 0 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: _check_auth_tickets 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f049c001690 con 0x7f04a00a98f0 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:30.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f0497fff640 10 monclient: dump: 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f04a76ae640 5 monclient: authenticate success, global_id 4279 2024-03-25T14:28:30.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.878+0000 7f04a76ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f049c0017f0 con 0x7f04a00a98f0 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f0497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f0497fff640 1 -- 172.21.15.26:0/1722443441 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f049c001c00 con 0x7f04a00a98f0 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f04a76ae640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f04a76ae640 4 set_mon_vals no callback set 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f0497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f0497fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.896+0000 7f0497fff640 10 monclient: dump: 2024-03-25T14:28:30.900 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:30.901 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 10 monclient: shutdown 2024-03-25T14:28:30.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:30.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 1 -- 172.21.15.26:0/1722443441 >> v1:172.21.15.121:6789/0 conn(0x7f04a00a98f0 legacy=0x7f04a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:30.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 1 -- 172.21.15.26:0/1722443441 shutdown_connections 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 1 -- 172.21.15.26:0/1722443441 >> 172.21.15.26:0/1722443441 conn(0x7f04a00b7ea0 msgr2=0x7f04a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 1 -- 172.21.15.26:0/1722443441 shutdown_connections 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.897+0000 7f04a76ae640 1 -- 172.21.15.26:0/1722443441 wait complete. 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 10 monclient: build_initial_monmap 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 10 monclient: monmap: 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:30.904 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 1 librados: starting msgr at 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 1 librados: starting objecter 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 1 Processor -- start 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 1 -- start start 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 1 librados: setting wanted keys 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 1 librados: calling monclient init 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.898+0000 7f04a76ae640 10 monclient: init 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient(hunting): picked mon.noname-b con 0x7f04a00a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient(hunting): picked mon.noname-a con 0x7f04a013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient(hunting): picked mon.noname-c con 0x7f04a01430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:30.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a013b2b0 con 0x7f04a013f710 2024-03-25T14:28:30.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a013b500 con 0x7f04a00a98f0 2024-03-25T14:28:30.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00b6f40 con 0x7f04a01430c0 2024-03-25T14:28:30.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:30.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.899+0000 7f04a76ae640 10 monclient(hunting): authenticate will time out at 1596.002116s 2024-03-25T14:28:30.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.915+0000 7f04a4c49640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f04a013f710 0x7f04a01df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44858/0 (socket says 172.21.15.26:44858) 2024-03-25T14:28:30.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.915+0000 7f04a4c49640 1 -- 172.21.15.26:0/3481183084 learned_addr learned my addr 172.21.15.26:0/3481183084 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:30.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.917+0000 7f0495ffb640 10 client.?.objecter ms_handle_connect 0x7f04a013f710 2024-03-25T14:28:30.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.917+0000 7f0495ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:30.925 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.921+0000 7f0495ffb640 10 client.?.objecter ms_handle_connect 0x7f04a00a98f0 2024-03-25T14:28:30.925 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.921+0000 7f0495ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:30.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915788197 0 0) 0x7f04a013b2b0 con 0x7f04a013f710 2024-03-25T14:28:30.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:30.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:30.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:30.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): my global_id is 4296 2024-03-25T14:28:30.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0484004eb0 con 0x7f04a013f710 2024-03-25T14:28:30.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830039928 0 0) 0x7f04a013b500 con 0x7f04a00a98f0 2024-03-25T14:28:30.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:30.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:30.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:30.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T14:28:30.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.927+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a013b2b0 con 0x7f04a00a98f0 2024-03-25T14:28:30.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.930+0000 7f0495ffb640 10 client.?.objecter ms_handle_connect 0x7f04a01430c0 2024-03-25T14:28:30.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.930+0000 7f0495ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:30.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.934+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521841307 0 0) 0x7f04a00b6f40 con 0x7f04a01430c0 2024-03-25T14:28:30.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.934+0000 7f0495ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:30.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.934+0000 7f0495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:30.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.934+0000 7f0495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:30.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.934+0000 7f0495ffb640 10 monclient(hunting): my global_id is 4280 2024-03-25T14:28:30.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.934+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a013b500 con 0x7f04a01430c0 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63367901 0 0) 0x7f0484004eb0 con 0x7f04a013f710 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a00b6f40 con 0x7f04a013f710 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0490001a10 con 0x7f04a013f710 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 10 monclient(hunting): dump: 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:30.942 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.938+0000 7f0495ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.941+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2344337264 0 0) 0x7f04a013b2b0 con 0x7f04a00a98f0 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.941+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0484004eb0 con 0x7f04a00a98f0 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.941+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f049c001bc0 con 0x7f04a00a98f0 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.941+0000 7f0495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.941+0000 7f0495ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.941+0000 7f0495ffb640 10 monclient(hunting): dump: 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:30.945 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:30.946 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.942+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1723838642 0 0) 0x7f04a013b500 con 0x7f04a01430c0 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.942+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a013b2b0 con 0x7f04a01430c0 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.942+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f04980019d0 con 0x7f04a01430c0 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.942+0000 7f0495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.942+0000 7f0495ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.942+0000 7f0495ffb640 10 monclient(hunting): dump: 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.947 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:30.948 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:30.948 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:30.948 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:30.948 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:30.948 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:30.948 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:30.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 136511827 0 0) 0x7f0484004eb0 con 0x7f04a00a98f0 2024-03-25T14:28:30.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 >> v1:172.21.15.26:6790/0 conn(0x7f04a01430c0 legacy=0x7f04a01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 >> v1:172.21.15.26:6789/0 conn(0x7f04a013f710 legacy=0x7f04a01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 1 monclient: found mon.b 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a013a430 con 0x7f04a00a98f0 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 10 monclient: _finish_auth 0 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f0495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:30.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 5 monclient: authenticate success, global_id 4282 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 10 monclient: _renew_subs 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a012dd10 con 0x7f04a00a98f0 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 10 monclient: _renew_subs 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a00b7b40 con 0x7f04a00a98f0 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.946+0000 7f04a76ae640 1 librados: init done 2024-03-25T14:28:30.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.947+0000 7f04a76ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:30.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.947+0000 7f04a76ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:30.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.947+0000 7f04a76ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:30.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:30.947+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a00627b0 con 0x7f04a00a98f0 2024-03-25T14:28:31.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f049c004a90 con 0x7f04a00a98f0 2024-03-25T14:28:31.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f0495ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:31.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f049c004da0 con 0x7f04a00a98f0 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f0495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f0495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f0495ffb640 10 monclient: dump: 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:31.231 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:31.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.226+0000 7f04977fe640 4 set_mon_vals no callback set 2024-03-25T14:28:31.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.227+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f049c01ad70 con 0x7f04a00a98f0 2024-03-25T14:28:31.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.227+0000 7f04a4c49640 10 monclient: get_auth_request con 0x7f048406e160 auth_method 0 2024-03-25T14:28:31.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.228+0000 7f0495ffb640 10 client.4282.objecter ms_handle_connect 0x7f048406e160 2024-03-25T14:28:31.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f049c0548c0 con 0x7f04a00a98f0 2024-03-25T14:28:31.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 10 client.4282.objecter ms_dispatch 0x7f04a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:31.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:31.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:31.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T14:28:31.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f049c0503d0 con 0x7f04a00a98f0 2024-03-25T14:28:31.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:31.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.231+0000 7f0495ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:31.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.494+0000 7f04a76ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:31.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.494+0000 7f04a76ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:31.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.494+0000 7f04a76ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:31.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.494+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04a01434a0 con 0x7f04a00a98f0 2024-03-25T14:28:31.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.546+0000 7f0495ffb640 1 -- 172.21.15.26:0/3481183084 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f049c052300 con 0x7f04a00a98f0 2024-03-25T14:28:31.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.546+0000 7f0495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:31.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.546+0000 7f0495ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:31.551 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:31.551 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 10 librados: watch_flush enter 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 10 librados: watch_flush exit 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 10 client.4282.objecter successfully canceled tick 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 >> v1:172.21.15.121:6800/288813066 conn(0x7f048406e160 legacy=0x7f0484070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 10 monclient: shutdown 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 >> v1:172.21.15.121:6789/0 conn(0x7f04a00a98f0 legacy=0x7f04a0139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 shutdown_connections 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 >> 172.21.15.26:0/3481183084 conn(0x7f04a00b7ea0 msgr2=0x7f04a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 shutdown_connections 2024-03-25T14:28:31.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 -- 172.21.15.26:0/3481183084 wait complete. 2024-03-25T14:28:31.554 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:31.548+0000 7f04a76ae640 1 librados: shutdown 2024-03-25T14:28:31.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:35.674 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:54.535 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.681 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:54.542 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.704 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:01:54.565 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.736 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:54.613 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.743 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:54.604 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.744 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:54.621 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.746 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:54.608 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.774 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:01:54.651 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:35.778 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:01:54.639 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:38.373 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:38.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.489+0000 7fc8b787b640 10 monclient: get_monmap_and_config 2024-03-25T14:28:38.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.489+0000 7fc8b787b640 10 monclient: build_initial_monmap 2024-03-25T14:28:38.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.489+0000 7fc8b787b640 10 monclient: monmap: 2024-03-25T14:28:38.493 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:38.493 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:38.493 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:38.490151+0000 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:38.490151+0000 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:38.494 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:38.495 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.489+0000 7fc8b787b640 1 Processor -- start 2024-03-25T14:28:38.495 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 1 -- start start 2024-03-25T14:28:38.495 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient: init 2024-03-25T14:28:38.495 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:38.495 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b00a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b01430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b0142520 con 0x7fc8b013f710 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b00b28a0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b00b2a30 con 0x7fc8b01430c0 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:38.496 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.490+0000 7fc8b787b640 10 monclient(hunting): authenticate will time out at 1603.593402s 2024-03-25T14:28:38.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.500+0000 7fc8b5e18640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc8b01430c0 0x7fc8b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:46290/0 (socket says 172.21.15.26:46290) 2024-03-25T14:28:38.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.500+0000 7fc8b5e18640 1 -- 172.21.15.26:0/4088318630 learned_addr learned my addr 172.21.15.26:0/4088318630 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:38.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.506+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658171565 0 0) 0x7fc8b00b2a30 con 0x7fc8b01430c0 2024-03-25T14:28:38.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.506+0000 7fc8a7fff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.506+0000 7fc8a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.506+0000 7fc8a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.506+0000 7fc8a7fff640 10 monclient(hunting): my global_id is 4283 2024-03-25T14:28:38.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.506+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c003520 con 0x7fc8b01430c0 2024-03-25T14:28:38.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2165902264 0 0) 0x7fc89c003520 con 0x7fc8b01430c0 2024-03-25T14:28:38.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 >> v1:172.21.15.121:6789/0 conn(0x7fc8b00a98f0 legacy=0x7fc8b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 >> v1:172.21.15.26:6789/0 conn(0x7fc8b013f710 legacy=0x7fc8b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: _finish_hunting 0 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 1 monclient: found mon.noname-b 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b00b44c0 con 0x7fc8b01430c0 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: _finish_auth 0 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: _check_auth_tickets 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc8a8001a10 con 0x7fc8b01430c0 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:38.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8a7fff640 10 monclient: dump: 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8b787b640 5 monclient: authenticate success, global_id 4283 2024-03-25T14:28:38.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.515+0000 7fc8b787b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:38.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8a8001b70 con 0x7fc8b01430c0 2024-03-25T14:28:38.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:38.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8a7fff640 1 -- 172.21.15.26:0/4088318630 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc8a8001fc0 con 0x7fc8b01430c0 2024-03-25T14:28:38.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8a7fff640 10 monclient: dump: 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:38.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8b787b640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:38.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.530+0000 7fc8b787b640 4 set_mon_vals no callback set 2024-03-25T14:28:38.537 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 10 monclient: shutdown 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 1 -- 172.21.15.26:0/4088318630 >> v1:172.21.15.26:6790/0 conn(0x7fc8b01430c0 legacy=0x7fc8b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 1 -- 172.21.15.26:0/4088318630 shutdown_connections 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 1 -- 172.21.15.26:0/4088318630 >> 172.21.15.26:0/4088318630 conn(0x7fc8b00b7ea0 msgr2=0x7fc8b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 1 -- 172.21.15.26:0/4088318630 shutdown_connections 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.531+0000 7fc8b787b640 1 -- 172.21.15.26:0/4088318630 wait complete. 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 10 monclient: build_initial_monmap 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 10 monclient: monmap: 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:38.538 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 1 librados: starting msgr at 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 1 librados: starting objecter 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 1 Processor -- start 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 1 -- start start 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 1 librados: setting wanted keys 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 1 librados: calling monclient init 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.532+0000 7fc8b787b640 10 monclient: init 2024-03-25T14:28:38.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b00a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b01430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b0136ec0 con 0x7fc8b013f710 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b0136c20 con 0x7fc8b00a98f0 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b0137db0 con 0x7fc8b01430c0 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:38.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.533+0000 7fc8b787b640 10 monclient(hunting): authenticate will time out at 1603.636079s 2024-03-25T14:28:38.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.548+0000 7fc8b5617640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc8b00a98f0 0x7fc8b01376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39024/0 (socket says 172.21.15.26:39024) 2024-03-25T14:28:38.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.548+0000 7fc8b5617640 1 -- 172.21.15.26:0/3753499272 learned_addr learned my addr 172.21.15.26:0/3753499272 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:38.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.549+0000 7fc8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc8b00a98f0 2024-03-25T14:28:38.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.549+0000 7fc8a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:38.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.551+0000 7fc8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc8b013f710 2024-03-25T14:28:38.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.551+0000 7fc8a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:38.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.561+0000 7fc8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc8b01430c0 2024-03-25T14:28:38.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.561+0000 7fc8a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:38.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.566+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247767703 0 0) 0x7fc8b0136c20 con 0x7fc8b00a98f0 2024-03-25T14:28:38.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.566+0000 7fc8a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.566+0000 7fc8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.566+0000 7fc8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.566+0000 7fc8a5ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T14:28:38.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.566+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc894004eb0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.570+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965191176 0 0) 0x7fc8b0136ec0 con 0x7fc8b013f710 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.570+0000 7fc8a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.575 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.570+0000 7fc8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.575 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.570+0000 7fc8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.575 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.570+0000 7fc8a5ffb640 10 monclient(hunting): my global_id is 4302 2024-03-25T14:28:38.575 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.570+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0136c20 con 0x7fc8b013f710 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720007271 0 0) 0x7fc8b0137db0 con 0x7fc8b01430c0 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): my global_id is 4286 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0136ec0 con 0x7fc8b01430c0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517363212 0 0) 0x7fc894004eb0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b0137db0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc8ac0019d0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): dump: 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.581+0000 7fc8a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2116816956 0 0) 0x7fc8b0137db0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 >> v1:172.21.15.26:6790/0 conn(0x7fc8b01430c0 legacy=0x7fc8b01e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 >> v1:172.21.15.26:6789/0 conn(0x7fc8b013f710 legacy=0x7fc8b00b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 1 monclient: found mon.b 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b0138b50 con 0x7fc8b00a98f0 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 10 monclient: _finish_auth 0 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8b787b640 5 monclient: authenticate success, global_id 4288 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8b787b640 10 monclient: _renew_subs 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8b787b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.592+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b0139830 con 0x7fc8b00a98f0 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.593+0000 7fc8b787b640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.593+0000 7fc8b787b640 10 monclient: _renew_subs 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.593+0000 7fc8b787b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.593+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b01369b0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.593+0000 7fc8b787b640 1 librados: init done 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.594+0000 7fc8b787b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.594+0000 7fc8b787b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.594+0000 7fc8b787b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.594+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b00627b0 con 0x7fc8b00a98f0 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8ac001c90 con 0x7fc8b00a98f0 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc8ac003a60 con 0x7fc8b00a98f0 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a5ffb640 10 monclient: dump: 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:38.873 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.874 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:38.874 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:38.874 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:38.874 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:38.874 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:38.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.868+0000 7fc8a77fe640 4 set_mon_vals no callback set 2024-03-25T14:28:38.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.869+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fc8ac003c80 con 0x7fc8b00a98f0 2024-03-25T14:28:38.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.870+0000 7fc8b4e16640 10 monclient: get_auth_request con 0x7fc89406de00 auth_method 0 2024-03-25T14:28:38.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.870+0000 7fc8a5ffb640 10 client.4288.objecter ms_handle_connect 0x7fc89406de00 2024-03-25T14:28:38.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.873+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fc8ac054810 con 0x7fc8b00a98f0 2024-03-25T14:28:38.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.873+0000 7fc8a5ffb640 10 client.4288.objecter ms_dispatch 0x7fc8b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:38.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.873+0000 7fc8a5ffb640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:38.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.873+0000 7fc8a5ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:38.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.873+0000 7fc8a5ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T14:28:38.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.876+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8ac050320 con 0x7fc8b00a98f0 2024-03-25T14:28:38.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.876+0000 7fc8a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:38.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:38.876+0000 7fc8a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:39.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.100+0000 7fc8b787b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:39.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.100+0000 7fc8b787b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:39.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.100+0000 7fc8b787b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:28:39.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.100+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8b0069f40 con 0x7fc8b00a98f0 2024-03-25T14:28:39.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.135+0000 7fc8a5ffb640 1 -- 172.21.15.26:0/3753499272 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fc8ac052250 con 0x7fc8b00a98f0 2024-03-25T14:28:39.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.135+0000 7fc8a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:39.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.135+0000 7fc8a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:39.140 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:39.140 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 10 librados: watch_flush enter 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 10 librados: watch_flush exit 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 10 client.4288.objecter successfully canceled tick 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 >> v1:172.21.15.121:6800/288813066 conn(0x7fc89406de00 legacy=0x7fc8940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 10 monclient: shutdown 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:39.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 >> v1:172.21.15.121:6789/0 conn(0x7fc8b00a98f0 legacy=0x7fc8b01376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:39.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 shutdown_connections 2024-03-25T14:28:39.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 >> 172.21.15.26:0/3753499272 conn(0x7fc8b00b7ea0 msgr2=0x7fc8b013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:39.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 shutdown_connections 2024-03-25T14:28:39.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.137+0000 7fc8b787b640 1 -- 172.21.15.26:0/3753499272 wait complete. 2024-03-25T14:28:39.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:39.138+0000 7fc8b787b640 1 librados: shutdown 2024-03-25T14:28:39.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:45.962 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:46.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.074+0000 7f820081e640 10 monclient: get_monmap_and_config 2024-03-25T14:28:46.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.074+0000 7f820081e640 10 monclient: build_initial_monmap 2024-03-25T14:28:46.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.074+0000 7f820081e640 10 monclient: monmap: 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:46.075175+0000 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:46.075175+0000 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:46.079 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 1 Processor -- start 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 1 -- start start 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient: init 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient(hunting): picked mon.noname-a con 0x7f81fc0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient(hunting): picked mon.noname-c con 0x7f81fc13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient(hunting): picked mon.noname-b con 0x7f81fc1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc142520 con 0x7f81fc0a98f0 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc0b28a0 con 0x7f81fc13f710 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc0b2a30 con 0x7f81fc1430c0 2024-03-25T14:28:46.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:46.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.075+0000 7f820081e640 10 monclient(hunting): authenticate will time out at 1611.178578s 2024-03-25T14:28:46.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.083+0000 7f81f9d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f81fc13f710 0x7f81fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:43810/0 (socket says 172.21.15.26:43810) 2024-03-25T14:28:46.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.083+0000 7f81f9d9b640 1 -- 172.21.15.26:0/2378445881 learned_addr learned my addr 172.21.15.26:0/2378445881 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:46.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.098+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452392769 0 0) 0x7f81fc0b28a0 con 0x7f81fc13f710 2024-03-25T14:28:46.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.098+0000 7f81f959a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.098+0000 7f81f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.098+0000 7f81f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.098+0000 7f81f959a640 10 monclient(hunting): my global_id is 4291 2024-03-25T14:28:46.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.098+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e4003520 con 0x7f81fc13f710 2024-03-25T14:28:46.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.112+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074443964 0 0) 0x7f81fc142520 con 0x7f81fc0a98f0 2024-03-25T14:28:46.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.112+0000 7f81f959a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.112+0000 7f81f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.112+0000 7f81f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.112+0000 7f81f959a640 10 monclient(hunting): my global_id is 4305 2024-03-25T14:28:46.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.112+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81fc0b28a0 con 0x7f81fc0a98f0 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472376224 0 0) 0x7f81e4003520 con 0x7f81fc13f710 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 >> v1:172.21.15.26:6790/0 conn(0x7f81fc1430c0 legacy=0x7f81fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 >> v1:172.21.15.26:6789/0 conn(0x7f81fc0a98f0 legacy=0x7f81fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: _finish_hunting 0 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 1 monclient: found mon.noname-c 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0b44c0 con 0x7f81fc13f710 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: _finish_auth 0 2024-03-25T14:28:46.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: _check_auth_tickets 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f81f0001690 con 0x7f81fc13f710 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f81f959a640 10 monclient: dump: 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:46.120 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:46.121 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:46.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f820081e640 5 monclient: authenticate success, global_id 4291 2024-03-25T14:28:46.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.114+0000 7f820081e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:46.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81f0001870 con 0x7f81fc13f710 2024-03-25T14:28:46.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f81f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:46.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f81f959a640 1 -- 172.21.15.26:0/2378445881 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f81f0001c80 con 0x7f81fc13f710 2024-03-25T14:28:46.138 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f81f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f81f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f81f959a640 10 monclient: dump: 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f820081e640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:46.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.134+0000 7f820081e640 4 set_mon_vals no callback set 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 10 monclient: shutdown 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 1 -- 172.21.15.26:0/2378445881 >> v1:172.21.15.121:6789/0 conn(0x7f81fc13f710 legacy=0x7f81fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 1 -- 172.21.15.26:0/2378445881 shutdown_connections 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 1 -- 172.21.15.26:0/2378445881 >> 172.21.15.26:0/2378445881 conn(0x7f81fc0b7ea0 msgr2=0x7f81fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 1 -- 172.21.15.26:0/2378445881 shutdown_connections 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.135+0000 7f820081e640 1 -- 172.21.15.26:0/2378445881 wait complete. 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.136+0000 7f820081e640 10 monclient: build_initial_monmap 2024-03-25T14:28:46.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.136+0000 7f820081e640 10 monclient: monmap: 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 1 librados: starting msgr at 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 1 librados: starting objecter 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 1 Processor -- start 2024-03-25T14:28:46.142 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 1 -- start start 2024-03-25T14:28:46.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 1 librados: setting wanted keys 2024-03-25T14:28:46.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 1 librados: calling monclient init 2024-03-25T14:28:46.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 10 monclient: init 2024-03-25T14:28:46.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:46.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:46.143 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 10 monclient(hunting): picked mon.noname-c con 0x7f81fc0a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.137+0000 7f820081e640 10 monclient(hunting): picked mon.noname-a con 0x7f81fc13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.138+0000 7f820081e640 10 monclient(hunting): picked mon.noname-b con 0x7f81fc1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.138+0000 7f820081e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc0abcc0 con 0x7f81fc13f710 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.138+0000 7f820081e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc0abfa0 con 0x7f81fc1430c0 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.138+0000 7f820081e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc139100 con 0x7f81fc0a98f0 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.138+0000 7f820081e640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:46.144 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.138+0000 7f820081e640 10 monclient(hunting): authenticate will time out at 1611.240640s 2024-03-25T14:28:46.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.141+0000 7f81f9d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f81fc13f710 0x7f81fc0ac3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47214/0 (socket says 172.21.15.26:47214) 2024-03-25T14:28:46.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.141+0000 7f81f9d9b640 1 -- 172.21.15.26:0/1937835191 learned_addr learned my addr 172.21.15.26:0/1937835191 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:46.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.142+0000 7f81eaffd640 10 client.?.objecter ms_handle_connect 0x7f81fc13f710 2024-03-25T14:28:46.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.143+0000 7f81eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:46.155 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 10 client.?.objecter ms_handle_connect 0x7f81fc1430c0 2024-03-25T14:28:46.155 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:46.155 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213876636 0 0) 0x7f81fc0abcc0 con 0x7f81fc13f710 2024-03-25T14:28:46.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T14:28:46.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.151+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc004eb0 con 0x7f81fc13f710 2024-03-25T14:28:46.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.158+0000 7f81eaffd640 10 client.?.objecter ms_handle_connect 0x7f81fc0a98f0 2024-03-25T14:28:46.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.158+0000 7f81eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:46.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.161+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933417163 0 0) 0x7f81fc0abfa0 con 0x7f81fc1430c0 2024-03-25T14:28:46.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.161+0000 7f81eaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.161+0000 7f81eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.161+0000 7f81eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.161+0000 7f81eaffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T14:28:46.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.161+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81fc0abcc0 con 0x7f81fc1430c0 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 259405586 0 0) 0x7f81dc004eb0 con 0x7f81fc13f710 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81fc0abfa0 con 0x7f81fc13f710 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f81f0001c40 con 0x7f81fc13f710 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 10 monclient(hunting): dump: 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.172 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:46.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.167+0000 7f81eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:46.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.172+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023910521 0 0) 0x7f81fc139100 con 0x7f81fc0a98f0 2024-03-25T14:28:46.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.172+0000 7f81eaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.172+0000 7f81eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.172+0000 7f81eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.172+0000 7f81eaffd640 10 monclient(hunting): my global_id is 4292 2024-03-25T14:28:46.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.172+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc004eb0 con 0x7f81fc0a98f0 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2589561586 0 0) 0x7f81fc0abfa0 con 0x7f81fc13f710 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 >> v1:172.21.15.26:6790/0 conn(0x7f81fc0a98f0 legacy=0x7f81fc138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 >> v1:172.21.15.121:6789/0 conn(0x7f81fc1430c0 legacy=0x7f81fc1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 1 monclient: found mon.a 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0acad0 con 0x7f81fc13f710 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 10 monclient: _finish_auth 0 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f81eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 5 monclient: authenticate success, global_id 4308 2024-03-25T14:28:46.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 10 monclient: _renew_subs 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc137cb0 con 0x7f81fc13f710 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 10 monclient: _renew_subs 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc1366b0 con 0x7f81fc13f710 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.175+0000 7f820081e640 1 librados: init done 2024-03-25T14:28:46.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.176+0000 7f820081e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:46.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.176+0000 7f820081e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:46.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.176+0000 7f820081e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:46.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.176+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc0627b0 con 0x7f81fc13f710 2024-03-25T14:28:46.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.180+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81f0001170 con 0x7f81fc13f710 2024-03-25T14:28:46.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.180+0000 7f81eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:46.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.180+0000 7f81f8d99640 4 set_mon_vals no callback set 2024-03-25T14:28:46.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.496+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f81f0004d90 con 0x7f81fc13f710 2024-03-25T14:28:46.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.496+0000 7f81eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.496+0000 7f81eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:46.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.496+0000 7f81eaffd640 10 monclient: dump: 2024-03-25T14:28:46.500 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:46.500 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:46.501 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:46.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.497+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f81f001ade0 con 0x7f81fc13f710 2024-03-25T14:28:46.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.497+0000 7f81fa59c640 10 monclient: get_auth_request con 0x7f81dc06dd80 auth_method 0 2024-03-25T14:28:46.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.498+0000 7f81eaffd640 10 client.4308.objecter ms_handle_connect 0x7f81dc06dd80 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f81f0054ad0 con 0x7f81fc13f710 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 10 client.4308.objecter ms_dispatch 0x7f81fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81f0050410 con 0x7f81fc13f710 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:46.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.504+0000 7f81eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:46.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.765+0000 7f820081e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:46.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.765+0000 7f820081e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:46.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.765+0000 7f820081e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:28:46.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.765+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81fc13cdd0 con 0x7f81fc13f710 2024-03-25T14:28:46.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.788+0000 7f81eaffd640 1 -- 172.21.15.26:0/1937835191 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f81f0052510 con 0x7f81fc13f710 2024-03-25T14:28:46.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.788+0000 7f81eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:46.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.788+0000 7f81eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:46.792 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:46.793 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 10 librados: watch_flush enter 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 10 librados: watch_flush exit 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 10 client.4308.objecter successfully canceled tick 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 >> v1:172.21.15.121:6800/288813066 conn(0x7f81dc06dd80 legacy=0x7f81dc070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 10 monclient: shutdown 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 >> v1:172.21.15.26:6789/0 conn(0x7f81fc13f710 legacy=0x7f81fc0ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 shutdown_connections 2024-03-25T14:28:46.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 >> 172.21.15.26:0/1937835191 conn(0x7f81fc0b7ea0 msgr2=0x7f81fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:46.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 shutdown_connections 2024-03-25T14:28:46.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.790+0000 7f820081e640 1 -- 172.21.15.26:0/1937835191 wait complete. 2024-03-25T14:28:46.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:46.791+0000 7f820081e640 1 librados: shutdown 2024-03-25T14:28:46.807 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:53.614 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:53.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.726+0000 7ff1376cf640 10 monclient: get_monmap_and_config 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.726+0000 7ff1376cf640 10 monclient: build_initial_monmap 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.726+0000 7ff1376cf640 10 monclient: monmap: 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:28:53.727249+0000 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:28:53.727249+0000 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:28:53.731 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:53.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.727+0000 7ff1376cf640 1 Processor -- start 2024-03-25T14:28:53.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.727+0000 7ff1376cf640 1 -- start start 2024-03-25T14:28:53.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.727+0000 7ff1376cf640 10 monclient: init 2024-03-25T14:28:53.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:53.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:53.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff13013fc60 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff130070570 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff1300727a0 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13013c9f0 con 0x7ff130070570 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13006fac0 con 0x7ff1300727a0 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1300715d0 con 0x7ff13013fc60 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:53.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.728+0000 7ff1376cf640 10 monclient(hunting): authenticate will time out at 1618.831155s 2024-03-25T14:28:53.744 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.740+0000 7ff134c6a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff130070570 0x7ff130072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37218/0 (socket says 172.21.15.26:37218) 2024-03-25T14:28:53.745 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.740+0000 7ff134c6a640 1 -- 172.21.15.26:0/1935396265 learned_addr learned my addr 172.21.15.26:0/1935396265 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:53.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.751+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608230243 0 0) 0x7ff13006fac0 con 0x7ff1300727a0 2024-03-25T14:28:53.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.751+0000 7ff127fff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.751+0000 7ff127fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.751+0000 7ff127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.751+0000 7ff127fff640 10 monclient(hunting): my global_id is 4297 2024-03-25T14:28:53.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.751+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c003520 con 0x7ff1300727a0 2024-03-25T14:28:53.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.757+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028746215 0 0) 0x7ff13013c9f0 con 0x7ff130070570 2024-03-25T14:28:53.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.757+0000 7ff127fff640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.757+0000 7ff127fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.757+0000 7ff127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.762 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.757+0000 7ff127fff640 10 monclient(hunting): my global_id is 4311 2024-03-25T14:28:53.762 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.757+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13006fac0 con 0x7ff130070570 2024-03-25T14:28:53.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.760+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568711533 0 0) 0x7ff11c003520 con 0x7ff1300727a0 2024-03-25T14:28:53.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.760+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 >> v1:172.21.15.26:6790/0 conn(0x7ff13013fc60 legacy=0x7ff13006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.760+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 >> v1:172.21.15.26:6789/0 conn(0x7ff130070570 legacy=0x7ff130072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: _finish_hunting 0 2024-03-25T14:28:53.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 1 monclient: found mon.noname-c 2024-03-25T14:28:53.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300718b0 con 0x7ff1300727a0 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: _finish_auth 0 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: _check_auth_tickets 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff128001a10 con 0x7ff1300727a0 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff127fff640 10 monclient: dump: 2024-03-25T14:28:53.766 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:53.767 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:53.768 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:53.768 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff1376cf640 5 monclient: authenticate success, global_id 4297 2024-03-25T14:28:53.768 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.761+0000 7ff1376cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff128001b70 con 0x7ff1300727a0 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff127fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff127fff640 1 -- 172.21.15.26:0/1935396265 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff128001f80 con 0x7ff1300727a0 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff1376cf640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff1376cf640 4 set_mon_vals no callback set 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff127fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.769+0000 7ff127fff640 10 monclient: dump: 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:53.774 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:53.775 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:53.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 10 monclient: shutdown 2024-03-25T14:28:53.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:53.777 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 1 -- 172.21.15.26:0/1935396265 >> v1:172.21.15.121:6789/0 conn(0x7ff1300727a0 legacy=0x7ff13014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 1 -- 172.21.15.26:0/1935396265 shutdown_connections 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 1 -- 172.21.15.26:0/1935396265 >> 172.21.15.26:0/1935396265 conn(0x7ff1300b4a80 msgr2=0x7ff1300b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 1 -- 172.21.15.26:0/1935396265 shutdown_connections 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.770+0000 7ff1376cf640 1 -- 172.21.15.26:0/1935396265 wait complete. 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.771+0000 7ff1376cf640 10 monclient: build_initial_monmap 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.771+0000 7ff1376cf640 10 monclient: monmap: 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:28:53.778 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.771+0000 7ff1376cf640 1 librados: starting msgr at 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.771+0000 7ff1376cf640 1 librados: starting objecter 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 Processor -- start 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 -- start start 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 librados: setting wanted keys 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 librados: calling monclient init 2024-03-25T14:28:53.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient: init 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff130070570 addr v1:172.21.15.26:6790/0 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff1300727a0 addr v1:172.21.15.26:6789/0 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff13013fc60 addr v1:172.21.15.121:6789/0 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13013b200 con 0x7ff1300727a0 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff130143a50 con 0x7ff13013fc60 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13013ace0 con 0x7ff130070570 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:53.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.772+0000 7ff1376cf640 10 monclient(hunting): authenticate will time out at 1618.875404s 2024-03-25T14:28:53.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.789+0000 7ff135c6c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff13013fc60 0x7ff1301e31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:40098/0 (socket says 172.21.15.26:40098) 2024-03-25T14:28:53.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.789+0000 7ff135c6c640 1 -- 172.21.15.26:0/159360908 learned_addr learned my addr 172.21.15.26:0/159360908 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:28:53.794 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.790+0000 7ff125ffb640 10 client.?.objecter ms_handle_connect 0x7ff13013fc60 2024-03-25T14:28:53.794 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.790+0000 7ff125ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:53.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.795+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257273756 0 0) 0x7ff130143a50 con 0x7ff13013fc60 2024-03-25T14:28:53.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.795+0000 7ff125ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.795+0000 7ff125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.795+0000 7ff125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.795+0000 7ff125ffb640 10 monclient(hunting): my global_id is 4300 2024-03-25T14:28:53.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.795+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff114004eb0 con 0x7ff13013fc60 2024-03-25T14:28:53.803 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.799+0000 7ff125ffb640 10 client.?.objecter ms_handle_connect 0x7ff1300727a0 2024-03-25T14:28:53.803 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.799+0000 7ff125ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:53.806 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.802+0000 7ff125ffb640 10 client.?.objecter ms_handle_connect 0x7ff130070570 2024-03-25T14:28:53.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.802+0000 7ff125ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:53.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.809+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584667162 0 0) 0x7ff13013ace0 con 0x7ff130070570 2024-03-25T14:28:53.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.809+0000 7ff125ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.809+0000 7ff125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.809+0000 7ff125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.809+0000 7ff125ffb640 10 monclient(hunting): my global_id is 4298 2024-03-25T14:28:53.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.809+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff130143a50 con 0x7ff130070570 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.814+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305978683 0 0) 0x7ff114004eb0 con 0x7ff13013fc60 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.815+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff13013ace0 con 0x7ff13013fc60 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.815+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff128001f40 con 0x7ff13013fc60 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.815+0000 7ff125ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.815+0000 7ff125ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.815+0000 7ff125ffb640 10 monclient(hunting): dump: 2024-03-25T14:28:53.819 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:53.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.815+0000 7ff125ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:53.824 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.820+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1430014854 0 0) 0x7ff130143a50 con 0x7ff130070570 2024-03-25T14:28:53.824 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.820+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff114004eb0 con 0x7ff130070570 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.820+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff12c0018a0 con 0x7ff130070570 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.820+0000 7ff125ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.820+0000 7ff125ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.820+0000 7ff125ffb640 10 monclient(hunting): dump: 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:53.825 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:53.828 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.824+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96730655 0 0) 0x7ff13013b200 con 0x7ff1300727a0 2024-03-25T14:28:53.828 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.824+0000 7ff125ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.824+0000 7ff125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.824+0000 7ff125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.824+0000 7ff125ffb640 10 monclient(hunting): my global_id is 4314 2024-03-25T14:28:53.829 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.824+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff130143a50 con 0x7ff1300727a0 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2969164968 0 0) 0x7ff114004eb0 con 0x7ff130070570 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 >> v1:172.21.15.121:6789/0 conn(0x7ff13013fc60 legacy=0x7ff1301e31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 >> v1:172.21.15.26:6789/0 conn(0x7ff1300727a0 legacy=0x7ff13014a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 1 monclient: found mon.c 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13013a820 con 0x7ff130070570 2024-03-25T14:28:53.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 10 monclient: _finish_auth 0 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff125ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 5 monclient: authenticate success, global_id 4298 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 10 monclient: _renew_subs 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13014b280 con 0x7ff130070570 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 10 monclient: _renew_subs 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1301db520 con 0x7ff130070570 2024-03-25T14:28:53.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.831+0000 7ff1376cf640 1 librados: init done 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.833+0000 7ff1376cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.833+0000 7ff1376cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.833+0000 7ff1376cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:53.833+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13013ae70 con 0x7ff130070570 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff12c001d50 con 0x7ff130070570 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff125ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff12c0037d0 con 0x7ff130070570 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff125ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff125ffb640 10 monclient: dump: 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:54.161 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:28:54.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.156+0000 7ff1277fe640 4 set_mon_vals no callback set 2024-03-25T14:28:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.157+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ff12c0039f0 con 0x7ff130070570 2024-03-25T14:28:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.158+0000 7ff134c6a640 10 monclient: get_auth_request con 0x7ff11406dee0 auth_method 0 2024-03-25T14:28:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.158+0000 7ff125ffb640 10 client.4298.objecter ms_handle_connect 0x7ff11406dee0 2024-03-25T14:28:54.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7ff12c053530 con 0x7ff130070570 2024-03-25T14:28:54.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 10 client.4298.objecter ms_dispatch 0x7ff130000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:54.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:54.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:54.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T14:28:54.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff12c04ee70 con 0x7ff130070570 2024-03-25T14:28:54.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:54.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.163+0000 7ff125ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:54.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.395+0000 7ff1376cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:54.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.395+0000 7ff1376cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:54.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.395+0000 7ff1376cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:28:54.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.395+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff130143be0 con 0x7ff130070570 2024-03-25T14:28:54.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.423+0000 7ff125ffb640 1 -- 172.21.15.26:0/159360908 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7ff12c050f70 con 0x7ff130070570 2024-03-25T14:28:54.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.423+0000 7ff125ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:54.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.423+0000 7ff125ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:54.428 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:28:54.428 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:54.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.425+0000 7ff1376cf640 10 librados: watch_flush enter 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 10 librados: watch_flush exit 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 10 client.4298.objecter successfully canceled tick 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 >> v1:172.21.15.121:6800/288813066 conn(0x7ff11406dee0 legacy=0x7ff1140703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 10 monclient: shutdown 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 >> v1:172.21.15.26:6790/0 conn(0x7ff130070570 legacy=0x7ff1301dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 shutdown_connections 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 >> 172.21.15.26:0/159360908 conn(0x7ff1300b4a80 msgr2=0x7ff1300ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 shutdown_connections 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 -- 172.21.15.26:0/159360908 wait complete. 2024-03-25T14:28:54.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:28:54.426+0000 7ff1376cf640 1 librados: shutdown 2024-03-25T14:28:54.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:55.711 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:02:14.572 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.772 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:02:14.633 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.813 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:02:14.690 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.819 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:02:14.696 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.819 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:02:14.681 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.822 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:02:14.683 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.837 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:02:14.698 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.844 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:02:14.721 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:55.867 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:02:14.729 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:01.250 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:01.364 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.360+0000 7f47c07c4640 10 monclient: get_monmap_and_config 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.360+0000 7f47c07c4640 10 monclient: build_initial_monmap 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.360+0000 7f47c07c4640 10 monclient: monmap: 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:01.361331+0000 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:01.361331+0000 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:01.365 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.361+0000 7f47c07c4640 1 Processor -- start 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.361+0000 7f47c07c4640 1 -- start start 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.361+0000 7f47c07c4640 10 monclient: init 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient(hunting): picked mon.noname-a con 0x7f47b80a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient(hunting): picked mon.noname-c con 0x7f47b813f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient(hunting): picked mon.noname-b con 0x7f47b81430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b8142520 con 0x7f47b80a98f0 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b80b28a0 con 0x7f47b813f710 2024-03-25T14:29:01.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b80b2a30 con 0x7f47b81430c0 2024-03-25T14:29:01.368 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:01.368 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.362+0000 7f47c07c4640 10 monclient(hunting): authenticate will time out at 1626.464780s 2024-03-25T14:29:01.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.382+0000 7f47bed61640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f47b81430c0 0x7f47b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40176/0 (socket says 172.21.15.26:40176) 2024-03-25T14:29:01.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.382+0000 7f47bed61640 1 -- 172.21.15.26:0/3340626877 learned_addr learned my addr 172.21.15.26:0/3340626877 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:01.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.393+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406101612 0 0) 0x7f47b8142520 con 0x7f47b80a98f0 2024-03-25T14:29:01.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.393+0000 7f47bd55e640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.393+0000 7f47bd55e640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.393+0000 7f47bd55e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.393+0000 7f47bd55e640 10 monclient(hunting): my global_id is 4317 2024-03-25T14:29:01.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.393+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47a4003520 con 0x7f47b80a98f0 2024-03-25T14:29:01.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.404+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282879632 0 0) 0x7f47b80b28a0 con 0x7f47b813f710 2024-03-25T14:29:01.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.404+0000 7f47bd55e640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.404+0000 7f47bd55e640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.404+0000 7f47bd55e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.404+0000 7f47bd55e640 10 monclient(hunting): my global_id is 4303 2024-03-25T14:29:01.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.404+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b8142520 con 0x7f47b813f710 2024-03-25T14:29:01.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.408+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188391738 0 0) 0x7f47b80b2a30 con 0x7f47b81430c0 2024-03-25T14:29:01.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.408+0000 7f47bd55e640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.408+0000 7f47bd55e640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.408+0000 7f47bd55e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.408+0000 7f47bd55e640 10 monclient(hunting): my global_id is 4301 2024-03-25T14:29:01.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.408+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b80b28a0 con 0x7f47b81430c0 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210234124 0 0) 0x7f47a4003520 con 0x7f47b80a98f0 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 >> v1:172.21.15.26:6790/0 conn(0x7f47b81430c0 legacy=0x7f47b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 >> v1:172.21.15.121:6789/0 conn(0x7f47b813f710 legacy=0x7f47b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: _finish_hunting 0 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 1 monclient: found mon.noname-a 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b80b44c0 con 0x7f47b80a98f0 2024-03-25T14:29:01.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: _finish_auth 0 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: _check_auth_tickets 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f47b4001690 con 0x7f47b80a98f0 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47bd55e640 10 monclient: dump: 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:01.416 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:01.417 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47c07c4640 5 monclient: authenticate success, global_id 4317 2024-03-25T14:29:01.418 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.410+0000 7f47c07c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47b40017f0 con 0x7f47b80a98f0 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47bd55e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47bd55e640 1 -- 172.21.15.26:0/3340626877 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f47b4001c00 con 0x7f47b80a98f0 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47c07c4640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47c07c4640 4 set_mon_vals no callback set 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47bd55e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47bd55e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.420+0000 7f47bd55e640 10 monclient: dump: 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.425 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.426 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:01.426 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:01.426 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:01.426 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:01.426 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 10 monclient: shutdown 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 1 -- 172.21.15.26:0/3340626877 >> v1:172.21.15.26:6789/0 conn(0x7f47b80a98f0 legacy=0x7f47b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 1 -- 172.21.15.26:0/3340626877 shutdown_connections 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 1 -- 172.21.15.26:0/3340626877 >> 172.21.15.26:0/3340626877 conn(0x7f47b80b7ea0 msgr2=0x7f47b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 1 -- 172.21.15.26:0/3340626877 shutdown_connections 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.421+0000 7f47c07c4640 1 -- 172.21.15.26:0/3340626877 wait complete. 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 10 monclient: build_initial_monmap 2024-03-25T14:29:01.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 10 monclient: monmap: 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 1 librados: starting msgr at 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 1 librados: starting objecter 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 1 Processor -- start 2024-03-25T14:29:01.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 1 -- start start 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 1 librados: setting wanted keys 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 1 librados: calling monclient init 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.422+0000 7f47c07c4640 10 monclient: init 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient(hunting): picked mon.noname-b con 0x7f47b80a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient(hunting): picked mon.noname-a con 0x7f47b813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient(hunting): picked mon.noname-c con 0x7f47b81430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b81db050 con 0x7f47b813f710 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b81dab70 con 0x7f47b80a98f0 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b81dad00 con 0x7f47b81430c0 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:01.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.423+0000 7f47c07c4640 10 monclient(hunting): authenticate will time out at 1626.526225s 2024-03-25T14:29:01.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.437+0000 7f47bed61640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f47b81430c0 0x7f47b81393b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40218/0 (socket says 172.21.15.26:40218) 2024-03-25T14:29:01.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.437+0000 7f47bed61640 1 -- 172.21.15.26:0/944324539 learned_addr learned my addr 172.21.15.26:0/944324539 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:01.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.439+0000 7f47aeffd640 10 client.?.objecter ms_handle_connect 0x7f47b81430c0 2024-03-25T14:29:01.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.439+0000 7f47aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:01.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.446+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372446756 0 0) 0x7f47b81dad00 con 0x7f47b81430c0 2024-03-25T14:29:01.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.446+0000 7f47aeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.446+0000 7f47aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.446+0000 7f47aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.446+0000 7f47aeffd640 10 monclient(hunting): my global_id is 4304 2024-03-25T14:29:01.451 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.446+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f479c004eb0 con 0x7f47b81430c0 2024-03-25T14:29:01.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.448+0000 7f47aeffd640 10 client.?.objecter ms_handle_connect 0x7f47b80a98f0 2024-03-25T14:29:01.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.448+0000 7f47aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:01.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.451+0000 7f47aeffd640 10 client.?.objecter ms_handle_connect 0x7f47b813f710 2024-03-25T14:29:01.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.451+0000 7f47aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:01.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3623190214 0 0) 0x7f479c004eb0 con 0x7f47b81430c0 2024-03-25T14:29:01.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47b81dad00 con 0x7f47b81430c0 2024-03-25T14:29:01.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f47b0001bd0 con 0x7f47b81430c0 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 10 monclient(hunting): dump: 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:01.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.458+0000 7f47aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:01.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.465+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271918639 0 0) 0x7f47b81db050 con 0x7f47b813f710 2024-03-25T14:29:01.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.465+0000 7f47aeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.465+0000 7f47aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.465+0000 7f47aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.465+0000 7f47aeffd640 10 monclient(hunting): my global_id is 4320 2024-03-25T14:29:01.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.465+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f479c004eb0 con 0x7f47b813f710 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3390599722 0 0) 0x7f47b81dad00 con 0x7f47b81430c0 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 >> v1:172.21.15.121:6789/0 conn(0x7f47b80a98f0 legacy=0x7f47b81db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 >> v1:172.21.15.26:6789/0 conn(0x7f47b813f710 legacy=0x7f47b81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 10 monclient: _finish_hunting 0 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 1 monclient: found mon.c 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b80b70a0 con 0x7f47b81430c0 2024-03-25T14:29:01.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 10 monclient: _finish_auth 0 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 10 monclient: _check_auth_tickets 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 5 monclient: authenticate success, global_id 4304 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 10 monclient: _renew_subs 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b80b72f0 con 0x7f47b81430c0 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 10 monclient: _renew_subs 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b81e32a0 con 0x7f47b81430c0 2024-03-25T14:29:01.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.466+0000 7f47c07c4640 1 librados: init done 2024-03-25T14:29:01.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.467+0000 7f47c07c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:01.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.467+0000 7f47c07c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:01.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.467+0000 7f47c07c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:29:01.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.467+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b80627b0 con 0x7f47b81430c0 2024-03-25T14:29:01.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47b00020c0 con 0x7f47b81430c0 2024-03-25T14:29:01.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f47b0004d00 con 0x7f47b81430c0 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47aeffd640 10 monclient: dump: 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:01.784 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:01.785 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.779+0000 7f47bcd5d640 4 set_mon_vals no callback set 2024-03-25T14:29:01.785 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.780+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f47b0004f20 con 0x7f47b81430c0 2024-03-25T14:29:01.785 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.780+0000 7f47be560640 10 monclient: get_auth_request con 0x7f479c06ddd0 auth_method 0 2024-03-25T14:29:01.785 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.781+0000 7f47aeffd640 10 client.4304.objecter ms_handle_connect 0x7f479c06ddd0 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f47b0054b80 con 0x7f47b81430c0 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 10 client.4304.objecter ms_dispatch 0x7f47b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47b00504c0 con 0x7f47b81430c0 2024-03-25T14:29:01.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:01.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:01.786+0000 7f47aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:02.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.045+0000 7f47c07c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:02.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.045+0000 7f47c07c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:02.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.045+0000 7f47c07c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:29:02.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.045+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47b81db200 con 0x7f47b81430c0 2024-03-25T14:29:02.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.068+0000 7f47aeffd640 1 -- 172.21.15.26:0/944324539 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f47b81db200 con 0x7f47b81430c0 2024-03-25T14:29:02.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.068+0000 7f47aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:02.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.068+0000 7f47aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:02.073 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:02.073 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:02.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 10 librados: watch_flush enter 2024-03-25T14:29:02.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 10 librados: watch_flush exit 2024-03-25T14:29:02.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 10 client.4304.objecter successfully canceled tick 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 >> v1:172.21.15.121:6800/288813066 conn(0x7f479c06ddd0 legacy=0x7f479c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 10 monclient: shutdown 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 >> v1:172.21.15.26:6790/0 conn(0x7f47b81430c0 legacy=0x7f47b81393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 shutdown_connections 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 >> 172.21.15.26:0/944324539 conn(0x7f47b80b7ea0 msgr2=0x7f47b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 shutdown_connections 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.070+0000 7f47c07c4640 1 -- 172.21.15.26:0/944324539 wait complete. 2024-03-25T14:29:02.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:02.071+0000 7f47c07c4640 1 librados: shutdown 2024-03-25T14:29:02.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:08.895 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.007+0000 7f649813b640 10 monclient: get_monmap_and_config 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.007+0000 7f649813b640 10 monclient: build_initial_monmap 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.007+0000 7f649813b640 10 monclient: monmap: 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:09.008491+0000 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:09.008491+0000 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.012 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:09.013 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:09.013 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:09.013 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.008+0000 7f649813b640 1 Processor -- start 2024-03-25T14:29:09.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.008+0000 7f649813b640 1 -- start start 2024-03-25T14:29:09.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.008+0000 7f649813b640 10 monclient: init 2024-03-25T14:29:09.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient(hunting): picked mon.noname-c con 0x7f64900a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient(hunting): picked mon.noname-b con 0x7f649013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient(hunting): picked mon.noname-a con 0x7f64901430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6490142520 con 0x7f64901430c0 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900b28a0 con 0x7f64900a98f0 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64900b2a30 con 0x7f649013f710 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:09.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.009+0000 7f649813b640 10 monclient(hunting): authenticate will time out at 1634.111993s 2024-03-25T14:29:09.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.024+0000 7f64966d8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f64901430c0 0x7f649014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:42778/0 (socket says 172.21.15.26:42778) 2024-03-25T14:29:09.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.024+0000 7f64966d8640 1 -- 172.21.15.26:0/2046170196 learned_addr learned my addr 172.21.15.26:0/2046170196 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:09.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.036+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470614302 0 0) 0x7f6490142520 con 0x7f64901430c0 2024-03-25T14:29:09.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.036+0000 7f6494ed5640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.036+0000 7f6494ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.036+0000 7f6494ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.036+0000 7f6494ed5640 10 monclient(hunting): my global_id is 4323 2024-03-25T14:29:09.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.036+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c003520 con 0x7f64901430c0 2024-03-25T14:29:09.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.046+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077053840 0 0) 0x7f64900b28a0 con 0x7f64900a98f0 2024-03-25T14:29:09.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.046+0000 7f6494ed5640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.046+0000 7f6494ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.046+0000 7f6494ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.046+0000 7f6494ed5640 10 monclient(hunting): my global_id is 4309 2024-03-25T14:29:09.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.046+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6490142520 con 0x7f64900a98f0 2024-03-25T14:29:09.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.049+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854944764 0 0) 0x7f64900b2a30 con 0x7f649013f710 2024-03-25T14:29:09.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.049+0000 7f6494ed5640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.049+0000 7f6494ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.049+0000 7f6494ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.049+0000 7f6494ed5640 10 monclient(hunting): my global_id is 4307 2024-03-25T14:29:09.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.049+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64900b28a0 con 0x7f649013f710 2024-03-25T14:29:09.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628416739 0 0) 0x7f64900b28a0 con 0x7f649013f710 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 >> v1:172.21.15.121:6789/0 conn(0x7f64900a98f0 legacy=0x7f64900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 >> v1:172.21.15.26:6789/0 conn(0x7f64901430c0 legacy=0x7f649014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: _finish_hunting 0 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 1 monclient: found mon.noname-b 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64900b44c0 con 0x7f649013f710 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: _finish_auth 0 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: _check_auth_tickets 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64800015b0 con 0x7f649013f710 2024-03-25T14:29:09.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f649813b640 5 monclient: authenticate success, global_id 4307 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f649813b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f6494ed5640 10 monclient: dump: 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:09.062 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.056+0000 7f649813b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:09.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6480001710 con 0x7f649013f710 2024-03-25T14:29:09.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f6494ed5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:09.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f6494ed5640 1 -- 172.21.15.26:0/2046170196 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6480001b60 con 0x7f649013f710 2024-03-25T14:29:09.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f649813b640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f649813b640 4 set_mon_vals no callback set 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f6494ed5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f6494ed5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f6494ed5640 10 monclient: dump: 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:09.073 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.068+0000 7f649813b640 10 monclient: shutdown 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 1 -- 172.21.15.26:0/2046170196 >> v1:172.21.15.26:6790/0 conn(0x7f649013f710 legacy=0x7f64901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 1 -- 172.21.15.26:0/2046170196 shutdown_connections 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 1 -- 172.21.15.26:0/2046170196 >> 172.21.15.26:0/2046170196 conn(0x7f64900b7ea0 msgr2=0x7f64900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 1 -- 172.21.15.26:0/2046170196 shutdown_connections 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 1 -- 172.21.15.26:0/2046170196 wait complete. 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 10 monclient: build_initial_monmap 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.069+0000 7f649813b640 10 monclient: monmap: 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:09.076 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 1 librados: starting msgr at 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 1 librados: starting objecter 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 1 Processor -- start 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 1 -- start start 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 1 librados: setting wanted keys 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 1 librados: calling monclient init 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 10 monclient: init 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:09.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.070+0000 7f649813b640 10 monclient(hunting): picked mon.noname-b con 0x7f64900a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 10 monclient(hunting): picked mon.noname-c con 0x7f649013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 10 monclient(hunting): picked mon.noname-a con 0x7f64901430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6490137c90 con 0x7f64901430c0 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6490139970 con 0x7f64900a98f0 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64901db7f0 con 0x7f649013f710 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:09.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.071+0000 7f649813b640 10 monclient(hunting): authenticate will time out at 1634.173702s 2024-03-25T14:29:09.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.078+0000 7f6495ed7640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f64900a98f0 0x7f64901db0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:38410/0 (socket says 172.21.15.26:38410) 2024-03-25T14:29:09.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.078+0000 7f6495ed7640 1 -- 172.21.15.26:0/1355171229 learned_addr learned my addr 172.21.15.26:0/1355171229 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:09.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.082+0000 7f64867fc640 10 client.?.objecter ms_handle_connect 0x7f64900a98f0 2024-03-25T14:29:09.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.082+0000 7f64867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:09.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.090+0000 7f64867fc640 10 client.?.objecter ms_handle_connect 0x7f649013f710 2024-03-25T14:29:09.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.090+0000 7f64867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:09.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.093+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582582226 0 0) 0x7f6490139970 con 0x7f64900a98f0 2024-03-25T14:29:09.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.093+0000 7f64867fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.093+0000 7f64867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.093+0000 7f64867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.093+0000 7f64867fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T14:29:09.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.093+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6474004eb0 con 0x7f64900a98f0 2024-03-25T14:29:09.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.104+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427285938 0 0) 0x7f64901db7f0 con 0x7f649013f710 2024-03-25T14:29:09.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.104+0000 7f64867fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.104+0000 7f64867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.104+0000 7f64867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.104+0000 7f64867fc640 10 monclient(hunting): my global_id is 4310 2024-03-25T14:29:09.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.104+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6490139970 con 0x7f649013f710 2024-03-25T14:29:09.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.107+0000 7f64867fc640 10 client.?.objecter ms_handle_connect 0x7f64901430c0 2024-03-25T14:29:09.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.107+0000 7f64867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:09.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.118+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052941239 0 0) 0x7f6490137c90 con 0x7f64901430c0 2024-03-25T14:29:09.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.118+0000 7f64867fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.118+0000 7f64867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.118+0000 7f64867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.118+0000 7f64867fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T14:29:09.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.118+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64901db7f0 con 0x7f64901430c0 2024-03-25T14:29:09.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 694637610 0 0) 0x7f6474004eb0 con 0x7f64900a98f0 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6490137c90 con 0x7f64900a98f0 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f648c001a10 con 0x7f64900a98f0 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 10 monclient(hunting): dump: 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.125 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:09.126 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:09.126 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:09.126 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.126 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.119+0000 7f64867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.130+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1004472772 0 0) 0x7f6490139970 con 0x7f649013f710 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.130+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6474004eb0 con 0x7f649013f710 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.130+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6480001a20 con 0x7f649013f710 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.130+0000 7f64867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.130+0000 7f64867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.130+0000 7f64867fc640 10 monclient(hunting): dump: 2024-03-25T14:29:09.135 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:09.136 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.140+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665763605 0 0) 0x7f64901db7f0 con 0x7f64901430c0 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.140+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6490139970 con 0x7f64901430c0 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.140+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64880019e0 con 0x7f64901430c0 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.140+0000 7f64867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.140+0000 7f64867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.140+0000 7f64867fc640 10 monclient(hunting): dump: 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.145 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:09.146 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:09.146 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:09.146 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1279577080 0 0) 0x7f6490137c90 con 0x7f64900a98f0 2024-03-25T14:29:09.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 >> v1:172.21.15.26:6790/0 conn(0x7f649013f710 legacy=0x7f6490138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 >> v1:172.21.15.26:6789/0 conn(0x7f64901430c0 legacy=0x7f64901e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 10 monclient: _finish_hunting 0 2024-03-25T14:29:09.147 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 1 monclient: found mon.b 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490137ee0 con 0x7f64900a98f0 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 10 monclient: _finish_auth 0 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 10 monclient: _check_auth_tickets 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f64867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 5 monclient: authenticate success, global_id 4312 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 10 monclient: _renew_subs 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6490137860 con 0x7f64900a98f0 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 10 monclient: _renew_subs 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:09.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6490138a60 con 0x7f64900a98f0 2024-03-25T14:29:09.149 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.141+0000 7f649813b640 1 librados: init done 2024-03-25T14:29:09.149 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.142+0000 7f649813b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:09.149 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.142+0000 7f649813b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:09.149 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.142+0000 7f649813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:09.149 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.142+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64900b3ef0 con 0x7f64900a98f0 2024-03-25T14:29:09.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f648c001c50 con 0x7f64900a98f0 2024-03-25T14:29:09.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f64867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:09.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f648c003a60 con 0x7f64900a98f0 2024-03-25T14:29:09.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f64867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f64867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:09.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f64867fc640 10 monclient: dump: 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:09.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.434+0000 7f6487fff640 4 set_mon_vals no callback set 2024-03-25T14:29:09.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.435+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f648c003c80 con 0x7f64900a98f0 2024-03-25T14:29:09.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.436+0000 7f64956d6640 10 monclient: get_auth_request con 0x7f647406e0f0 auth_method 0 2024-03-25T14:29:09.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.436+0000 7f64867fc640 10 client.4312.objecter ms_handle_connect 0x7f647406e0f0 2024-03-25T14:29:09.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.442+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f648c054820 con 0x7f64900a98f0 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.442+0000 7f64867fc640 10 client.4312.objecter ms_dispatch 0x7f6490000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.442+0000 7f64867fc640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.442+0000 7f64867fc640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.442+0000 7f64867fc640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.443+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f648c050330 con 0x7f64900a98f0 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.443+0000 7f64867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.443+0000 7f64867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.705+0000 7f649813b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.705+0000 7f649813b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.705+0000 7f649813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.705+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64900627b0 con 0x7f64900a98f0 2024-03-25T14:29:09.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.761+0000 7f64867fc640 1 -- 172.21.15.26:0/1355171229 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f648c052260 con 0x7f64900a98f0 2024-03-25T14:29:09.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.761+0000 7f64867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:09.766 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.761+0000 7f64867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:09.766 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:09.766 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 10 librados: watch_flush enter 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 10 librados: watch_flush exit 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 10 client.4312.objecter successfully canceled tick 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 >> v1:172.21.15.121:6800/288813066 conn(0x7f647406e0f0 legacy=0x7f64740705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 10 monclient: shutdown 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 >> v1:172.21.15.121:6789/0 conn(0x7f64900a98f0 legacy=0x7f64901db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 shutdown_connections 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 >> 172.21.15.26:0/1355171229 conn(0x7f64900b7ea0 msgr2=0x7f6490074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:09.769 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 shutdown_connections 2024-03-25T14:29:09.770 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 -- 172.21.15.26:0/1355171229 wait complete. 2024-03-25T14:29:09.770 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:09.764+0000 7f649813b640 1 librados: shutdown 2024-03-25T14:29:09.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:16.589 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.673+0000 7f5178a6b640 10 monclient: get_monmap_and_config 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.673+0000 7f5178a6b640 10 monclient: build_initial_monmap 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.673+0000 7f5178a6b640 10 monclient: monmap: 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:16.674409+0000 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:16.674409+0000 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:16.678 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:16.679 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:16.679 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:16.679 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:16.679 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:16.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 1 Processor -- start 2024-03-25T14:29:16.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 1 -- start start 2024-03-25T14:29:16.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 10 monclient: init 2024-03-25T14:29:16.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:16.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:29:16.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f51740b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f51740b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f517414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740af940 con 0x7f517414be90 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.674+0000 7f5178a6b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740b0a70 con 0x7f51740b1730 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.675+0000 7f5178a6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740b0d50 con 0x7f51740b4d60 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.675+0000 7f5178a6b640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.675+0000 7f5178a6b640 10 monclient(hunting): authenticate will time out at 1641.777609s 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.675+0000 7f5172d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f517414be90 0x7f517414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46016/0 (socket says 172.21.15.26:46016) 2024-03-25T14:29:16.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:16.675+0000 7f5172d9d640 1 -- 172.21.15.26:0/4238232899 learned_addr learned my addr 172.21.15.26:0/4238232899 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): tick 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 monclient(hunting): continuing hunt 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.26:6790/0 conn(0x7f51740b4d60 legacy=0x7f517413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.121:6789/0 conn(0x7f51740b1730 legacy=0x7f51740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.26:6789/0 conn(0x7f517414be90 legacy=0x7f517414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): picked mon.noname-a con 0x7f51540036c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): picked mon.noname-c con 0x7f51740b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:19.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): picked mon.noname-b con 0x7f51740b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:19.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740b1030 con 0x7f51540036c0 2024-03-25T14:29:19.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51740b0d50 con 0x7f51740b1730 2024-03-25T14:29:19.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154010420 con 0x7f51740b4d60 2024-03-25T14:29:19.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:19.675+0000 7f5170d99640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:22.679 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): tick 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 monclient(hunting): continuing hunt 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.26:6790/0 conn(0x7f51740b4d60 legacy=0x7f515400cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.121:6789/0 conn(0x7f51740b1730 legacy=0x7f51540093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.26:6789/0 conn(0x7f51540036c0 legacy=0x7f5154005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): picked mon.noname-c con 0x7f5154010730 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): picked mon.noname-b con 0x7f515400f3d0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): picked mon.noname-a con 0x7f515400ff60 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154011680 con 0x7f515400ff60 2024-03-25T14:29:22.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154010420 con 0x7f5154010730 2024-03-25T14:29:22.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154006360 con 0x7f515400f3d0 2024-03-25T14:29:22.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:22.674+0000 7f5170d99640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:24.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:24.985+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073153655 0 0) 0x7f5154011680 con 0x7f515400ff60 2024-03-25T14:29:24.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:24.985+0000 7f5171d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:24.985+0000 7f5171d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:24.985+0000 7f5171d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:24.985+0000 7f5171d9b640 10 monclient(hunting): my global_id is 4329 2024-03-25T14:29:24.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:24.985+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5160003520 con 0x7f515400ff60 2024-03-25T14:29:25.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.014+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1203662999 0 0) 0x7f5160003520 con 0x7f515400ff60 2024-03-25T14:29:25.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.014+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.26:6790/0 conn(0x7f515400f3d0 legacy=0x7f515400f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:29:25.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.121:6789/0 conn(0x7f5154010730 legacy=0x7f5154010b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: _finish_hunting 0 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 1 monclient: found mon.noname-a 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154006540 con 0x7f515400ff60 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: _finish_auth 0 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: _check_auth_tickets 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f516c0019d0 con 0x7f515400ff60 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5178a6b640 5 monclient: authenticate success, global_id 4329 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5178a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:25.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5171d9b640 10 monclient: dump: 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:25.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.015+0000 7f5178a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.033+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f516c001ba0 con 0x7f515400ff60 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.033+0000 7f5171d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.033+0000 7f5171d9b640 1 -- 172.21.15.26:0/4238232899 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f516c001f30 con 0x7f515400ff60 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.033+0000 7f5171d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5171d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5171d9b640 10 monclient: dump: 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:25.039 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:25.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 4 set_mon_vals no callback set 2024-03-25T14:29:25.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 10 monclient: shutdown 2024-03-25T14:29:25.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:25.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 1 -- 172.21.15.26:0/4238232899 >> v1:172.21.15.26:6789/0 conn(0x7f515400ff60 legacy=0x7f5154018930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 1 -- 172.21.15.26:0/4238232899 shutdown_connections 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 1 -- 172.21.15.26:0/4238232899 >> 172.21.15.26:0/4238232899 conn(0x7f51740617a0 msgr2=0x7f5174061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 1 -- 172.21.15.26:0/4238232899 shutdown_connections 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.034+0000 7f5178a6b640 1 -- 172.21.15.26:0/4238232899 wait complete. 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.035+0000 7f5178a6b640 10 monclient: build_initial_monmap 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.035+0000 7f5178a6b640 10 monclient: monmap: 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:25.043 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 1 librados: starting msgr at 2024-03-25T14:29:25.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 1 librados: starting objecter 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 1 Processor -- start 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 1 -- start start 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 1 librados: setting wanted keys 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 1 librados: calling monclient init 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 10 monclient: init 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.036+0000 7f5178a6b640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f515400f3d0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:25.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f515400ff60 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:25.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f5154010730 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:25.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51741d8740 con 0x7f515400ff60 2024-03-25T14:29:25.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51741d7830 con 0x7f515400f3d0 2024-03-25T14:29:25.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51741d7b10 con 0x7f5154010730 2024-03-25T14:29:25.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:25.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.037+0000 7f5178a6b640 10 monclient(hunting): authenticate will time out at 1650.140814s 2024-03-25T14:29:25.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.038+0000 7f516bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f515400ff60 0x7f517413a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56072/0 (socket says 172.21.15.26:56072) 2024-03-25T14:29:25.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.038+0000 7f516bfff640 1 -- 172.21.15.26:0/352323541 learned_addr learned my addr 172.21.15.26:0/352323541 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:25.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.039+0000 7f516affd640 10 client.?.objecter ms_handle_connect 0x7f515400ff60 2024-03-25T14:29:25.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.039+0000 7f516affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:25.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.044+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164086308 0 0) 0x7f51741d8740 con 0x7f515400ff60 2024-03-25T14:29:25.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.044+0000 7f516affd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:25.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.044+0000 7f516affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:25.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.044+0000 7f516affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:25.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.044+0000 7f516affd640 10 monclient(hunting): my global_id is 4332 2024-03-25T14:29:25.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.044+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51540033f0 con 0x7f515400ff60 2024-03-25T14:29:25.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 10 client.?.objecter ms_handle_connect 0x7f515400f3d0 2024-03-25T14:29:25.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:25.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 522209419 0 0) 0x7f51540033f0 con 0x7f515400ff60 2024-03-25T14:29:25.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51741d8740 con 0x7f515400ff60 2024-03-25T14:29:25.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f515c001840 con 0x7f515400ff60 2024-03-25T14:29:25.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:25.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:25.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 10 monclient(hunting): dump: 2024-03-25T14:29:25.087 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:25.088 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.080+0000 7f516affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:25.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.161+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2204479662 0 0) 0x7f51741d8740 con 0x7f515400ff60 2024-03-25T14:29:25.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 >> v1:172.21.15.26:6790/0 conn(0x7f5154010730 legacy=0x7f517412ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 >> v1:172.21.15.121:6789/0 conn(0x7f515400f3d0 legacy=0x7f51741d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 10 monclient: _finish_hunting 0 2024-03-25T14:29:25.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 1 monclient: found mon.a 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517413a750 con 0x7f515400ff60 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 10 monclient: _finish_auth 0 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 10 monclient: _check_auth_tickets 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f516affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 5 monclient: authenticate success, global_id 4332 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 10 monclient: _renew_subs 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51741306c0 con 0x7f515400ff60 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 10 monclient: _renew_subs 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5174139ad0 con 0x7f515400ff60 2024-03-25T14:29:25.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.162+0000 7f5178a6b640 1 librados: init done 2024-03-25T14:29:25.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.163+0000 7f5178a6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:25.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.163+0000 7f5178a6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:25.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.163+0000 7f5178a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:29:25.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.163+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51741d7ca0 con 0x7f515400ff60 2024-03-25T14:29:25.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.167+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f515c0019a0 con 0x7f515400ff60 2024-03-25T14:29:25.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.167+0000 7f516affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:25.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.167+0000 7f517159a640 4 set_mon_vals no callback set 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.442+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f515c0038d0 con 0x7f515400ff60 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.443+0000 7f516affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.443+0000 7f516affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.443+0000 7f516affd640 10 monclient: dump: 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.448 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:25.449 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.443+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f515c003ab0 con 0x7f515400ff60 2024-03-25T14:29:25.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.444+0000 7f517259c640 10 monclient: get_auth_request con 0x7f515406f5a0 auth_method 0 2024-03-25T14:29:25.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.444+0000 7f516affd640 10 client.4332.objecter ms_handle_connect 0x7f515406f5a0 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f515c054840 con 0x7f515400ff60 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 10 client.4332.objecter ms_dispatch 0x7f5174000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f515c050350 con 0x7f515400ff60 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:25.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.449+0000 7f516affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:25.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.683+0000 7f5178a6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:25.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.683+0000 7f5178a6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:25.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.683+0000 7f5178a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:29:25.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.683+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51740627b0 con 0x7f515400ff60 2024-03-25T14:29:25.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.725+0000 7f516affd640 1 -- 172.21.15.26:0/352323541 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f515c052280 con 0x7f515400ff60 2024-03-25T14:29:25.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.725+0000 7f516affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:25.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.725+0000 7f516affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:25.731 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:25.731 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 10 librados: watch_flush enter 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 10 librados: watch_flush exit 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 10 client.4332.objecter successfully canceled tick 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 >> v1:172.21.15.121:6800/288813066 conn(0x7f515406f5a0 legacy=0x7f5154071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 10 monclient: shutdown 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 >> v1:172.21.15.26:6789/0 conn(0x7f515400ff60 legacy=0x7f517413a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 shutdown_connections 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 >> 172.21.15.26:0/352323541 conn(0x7f51740617a0 msgr2=0x7f51740b6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:25.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 shutdown_connections 2024-03-25T14:29:25.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 -- 172.21.15.26:0/352323541 wait complete. 2024-03-25T14:29:25.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:25.728+0000 7f5178a6b640 1 librados: shutdown 2024-03-25T14:29:25.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:30.797 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:02:49.657 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:30.849 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:02:49.709 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:30.879 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:02:49.740 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:30.931 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:02:49.807 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:30.933 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:02:49.810 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:30.960 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:02:49.837 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.552 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:32.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.652+0000 7efd7f7c8640 10 monclient: get_monmap_and_config 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.652+0000 7efd7f7c8640 10 monclient: build_initial_monmap 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.652+0000 7efd7f7c8640 10 monclient: monmap: 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:32.654331+0000 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:32.654331+0000 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:32.658 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.653+0000 7efd7f7c8640 1 Processor -- start 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.653+0000 7efd7f7c8640 1 -- start start 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.653+0000 7efd7f7c8640 10 monclient: init 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient(hunting): picked mon.noname-b con 0x7efd780b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient(hunting): picked mon.noname-a con 0x7efd780b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient(hunting): picked mon.noname-c con 0x7efd7814be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780af940 con 0x7efd780b1730 2024-03-25T14:29:32.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780b0a70 con 0x7efd7814be90 2024-03-25T14:29:32.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780b0d50 con 0x7efd780b4d60 2024-03-25T14:29:32.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:32.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.654+0000 7efd7f7c8640 10 monclient(hunting): authenticate will time out at 1657.758093s 2024-03-25T14:29:32.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.667+0000 7efd7dd65640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efd7814be90 0x7efd7814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:37376/0 (socket says 172.21.15.26:37376) 2024-03-25T14:29:32.672 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.667+0000 7efd7dd65640 1 -- 172.21.15.26:0/1683473046 learned_addr learned my addr 172.21.15.26:0/1683473046 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:32.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.679+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516541360 0 0) 0x7efd780af940 con 0x7efd780b1730 2024-03-25T14:29:32.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.680+0000 7efd6ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.680+0000 7efd6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.680+0000 7efd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.680+0000 7efd6ffff640 10 monclient(hunting): my global_id is 4335 2024-03-25T14:29:32.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.680+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd64003520 con 0x7efd780b1730 2024-03-25T14:29:32.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.682+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862843024 0 0) 0x7efd780b0a70 con 0x7efd7814be90 2024-03-25T14:29:32.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.683+0000 7efd6ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.683+0000 7efd6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.683+0000 7efd6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.683+0000 7efd6ffff640 10 monclient(hunting): my global_id is 4321 2024-03-25T14:29:32.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.683+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd780af940 con 0x7efd7814be90 2024-03-25T14:29:32.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2962123968 0 0) 0x7efd64003520 con 0x7efd780b1730 2024-03-25T14:29:32.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 >> v1:172.21.15.26:6790/0 conn(0x7efd780b4d60 legacy=0x7efd7813f660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T14:29:32.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 >> v1:172.21.15.121:6789/0 conn(0x7efd7814be90 legacy=0x7efd7814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 1 monclient: found mon.noname-a 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780b1030 con 0x7efd780b1730 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 10 monclient: _finish_auth 0 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd7f7c8640 5 monclient: authenticate success, global_id 4335 2024-03-25T14:29:32.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.684+0000 7efd7f7c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.686+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efd68001650 con 0x7efd780b1730 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.686+0000 7efd6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.686+0000 7efd6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.686+0000 7efd6ffff640 10 monclient: dump: 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:32.691 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:32.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.686+0000 7efd7f7c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:32.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd680018a0 con 0x7efd780b1730 2024-03-25T14:29:32.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:32.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd6ffff640 1 -- 172.21.15.26:0/1683473046 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efd68001c30 con 0x7efd780b1730 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd6ffff640 10 monclient: dump: 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:32.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd7f7c8640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:32.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.695+0000 7efd7f7c8640 4 set_mon_vals no callback set 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 10 monclient: shutdown 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 1 -- 172.21.15.26:0/1683473046 >> v1:172.21.15.26:6789/0 conn(0x7efd780b1730 legacy=0x7efd780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 1 -- 172.21.15.26:0/1683473046 shutdown_connections 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 1 -- 172.21.15.26:0/1683473046 >> 172.21.15.26:0/1683473046 conn(0x7efd780617a0 msgr2=0x7efd78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 1 -- 172.21.15.26:0/1683473046 shutdown_connections 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.696+0000 7efd7f7c8640 1 -- 172.21.15.26:0/1683473046 wait complete. 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 10 monclient: build_initial_monmap 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 10 monclient: monmap: 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:32.704 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 1 librados: starting msgr at 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 1 librados: starting objecter 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 1 Processor -- start 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 1 -- start start 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 1 librados: setting wanted keys 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 1 librados: calling monclient init 2024-03-25T14:29:32.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.697+0000 7efd7f7c8640 10 monclient: init 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient(hunting): picked mon.noname-c con 0x7efd780b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient(hunting): picked mon.noname-a con 0x7efd780b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient(hunting): picked mon.noname-b con 0x7efd7814be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd78141f10 con 0x7efd780b4d60 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd781daef0 con 0x7efd7814be90 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7813bd10 con 0x7efd780b1730 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:32.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.698+0000 7efd7f7c8640 10 monclient(hunting): authenticate will time out at 1657.801858s 2024-03-25T14:29:32.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.705+0000 7efd7dd65640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efd7814be90 0x7efd781e21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:37434/0 (socket says 172.21.15.26:37434) 2024-03-25T14:29:32.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.705+0000 7efd7dd65640 1 -- 172.21.15.26:0/3856562946 learned_addr learned my addr 172.21.15.26:0/3856562946 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:32.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.706+0000 7efd6dffb640 10 client.?.objecter ms_handle_connect 0x7efd7814be90 2024-03-25T14:29:32.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.706+0000 7efd6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.724+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910115340 0 0) 0x7efd781daef0 con 0x7efd7814be90 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.724+0000 7efd6dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.724+0000 7efd6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.724+0000 7efd6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.724+0000 7efd6dffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.724+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5c004eb0 con 0x7efd7814be90 2024-03-25T14:29:32.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.725+0000 7efd6dffb640 10 client.?.objecter ms_handle_connect 0x7efd780b4d60 2024-03-25T14:29:32.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.725+0000 7efd6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:32.741 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793692399 0 0) 0x7efd5c004eb0 con 0x7efd7814be90 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd781daef0 con 0x7efd7814be90 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efd70001c10 con 0x7efd7814be90 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 10 monclient(hunting): dump: 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:32.742 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:32.743 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:32.743 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:32.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.736+0000 7efd6dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:32.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.743+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874448497 0 0) 0x7efd78141f10 con 0x7efd780b4d60 2024-03-25T14:29:32.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.743+0000 7efd6dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.743+0000 7efd6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.743+0000 7efd6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.743+0000 7efd6dffb640 10 monclient(hunting): my global_id is 4338 2024-03-25T14:29:32.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.743+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5c004eb0 con 0x7efd780b4d60 2024-03-25T14:29:32.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 558804776 0 0) 0x7efd781daef0 con 0x7efd7814be90 2024-03-25T14:29:32.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 >> v1:172.21.15.26:6790/0 conn(0x7efd780b1730 legacy=0x7efd7813b600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-25T14:29:32.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 >> v1:172.21.15.26:6789/0 conn(0x7efd780b4d60 legacy=0x7efd781db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 10 monclient: _finish_hunting 0 2024-03-25T14:29:32.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 1 monclient: found mon.b 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd781dbb20 con 0x7efd7814be90 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 10 monclient: _finish_auth 0 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 10 monclient: _check_auth_tickets 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd7f7c8640 5 monclient: authenticate success, global_id 4324 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd7f7c8640 10 monclient: _renew_subs 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd7f7c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.744+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd781405f0 con 0x7efd7814be90 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.745+0000 7efd7f7c8640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.745+0000 7efd7f7c8640 10 monclient: _renew_subs 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.745+0000 7efd7f7c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:32.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.745+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd781e3900 con 0x7efd7814be90 2024-03-25T14:29:32.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.745+0000 7efd7f7c8640 1 librados: init done 2024-03-25T14:29:32.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.746+0000 7efd7f7c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:32.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.746+0000 7efd7f7c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:32.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.746+0000 7efd7f7c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:32.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:32.746+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd781401f0 con 0x7efd7814be90 2024-03-25T14:29:33.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd70004b60 con 0x7efd7814be90 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7efd70004e70 con 0x7efd7814be90 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6dffb640 10 monclient: dump: 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:33.027 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.021+0000 7efd6f7fe640 4 set_mon_vals no callback set 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.022+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7efd7001ae40 con 0x7efd7814be90 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.023+0000 7efd7d564640 10 monclient: get_auth_request con 0x7efd5c06dc90 auth_method 0 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.023+0000 7efd6dffb640 10 client.4324.objecter ms_handle_connect 0x7efd5c06dc90 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7efd70054b00 con 0x7efd7814be90 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 10 client.4324.objecter ms_dispatch 0x7efd78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd70050440 con 0x7efd7814be90 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:33.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.028+0000 7efd6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:33.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.250+0000 7efd7f7c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:33.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.250+0000 7efd7f7c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:33.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.250+0000 7efd7f7c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:33.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.250+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd780627b0 con 0x7efd7814be90 2024-03-25T14:29:33.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.276+0000 7efd6dffb640 1 -- 172.21.15.26:0/3856562946 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7efd70052540 con 0x7efd7814be90 2024-03-25T14:29:33.282 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.276+0000 7efd6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:33.282 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.276+0000 7efd6dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:33.282 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:33.282 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:33.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.278+0000 7efd7f7c8640 10 librados: watch_flush enter 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.278+0000 7efd7f7c8640 10 librados: watch_flush exit 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.278+0000 7efd7f7c8640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.278+0000 7efd7f7c8640 10 client.4324.objecter successfully canceled tick 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 >> v1:172.21.15.121:6800/288813066 conn(0x7efd5c06dc90 legacy=0x7efd5c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 10 monclient: shutdown 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 >> v1:172.21.15.121:6789/0 conn(0x7efd7814be90 legacy=0x7efd781e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 shutdown_connections 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 >> 172.21.15.26:0/3856562946 conn(0x7efd780617a0 msgr2=0x7efd780b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 shutdown_connections 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 -- 172.21.15.26:0/3856562946 wait complete. 2024-03-25T14:29:33.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:33.279+0000 7efd7f7c8640 1 librados: shutdown 2024-03-25T14:29:33.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:40.103 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:40.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.204+0000 7ff143f54640 10 monclient: get_monmap_and_config 2024-03-25T14:29:40.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.204+0000 7ff143f54640 10 monclient: build_initial_monmap 2024-03-25T14:29:40.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.204+0000 7ff143f54640 10 monclient: monmap: 2024-03-25T14:29:40.208 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:40.208 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:40.205024+0000 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:40.205024+0000 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:40.209 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:40.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.204+0000 7ff143f54640 1 Processor -- start 2024-03-25T14:29:40.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.205+0000 7ff143f54640 1 -- start start 2024-03-25T14:29:40.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.205+0000 7ff143f54640 10 monclient: init 2024-03-25T14:29:40.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.205+0000 7ff143f54640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:40.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.205+0000 7ff143f54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.205+0000 7ff143f54640 10 monclient(hunting): picked mon.noname-c con 0x7ff13c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 10 monclient(hunting): picked mon.noname-b con 0x7ff13c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 10 monclient(hunting): picked mon.noname-a con 0x7ff13c1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c142520 con 0x7ff13c1430c0 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c0b28a0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c0b2a30 con 0x7ff13c13f710 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:40.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.206+0000 7ff143f54640 10 monclient(hunting): authenticate will time out at 1665.308686s 2024-03-25T14:29:40.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.217+0000 7ff141cf0640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff13c0a98f0 0x7ff13c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:46046/0 (socket says 172.21.15.26:46046) 2024-03-25T14:29:40.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.217+0000 7ff141cf0640 1 -- 172.21.15.26:0/1787337469 learned_addr learned my addr 172.21.15.26:0/1787337469 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:40.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.226+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450941782 0 0) 0x7ff13c0b28a0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.226+0000 7ff140cee640 10 monclient(hunting): _negotiate 2024-03-25T14:29:40.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.226+0000 7ff140cee640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:40.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.226+0000 7ff140cee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:40.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.226+0000 7ff140cee640 10 monclient(hunting): my global_id is 4327 2024-03-25T14:29:40.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.226+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff128003520 con 0x7ff13c0a98f0 2024-03-25T14:29:40.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.234+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579595901 0 0) 0x7ff13c142520 con 0x7ff13c1430c0 2024-03-25T14:29:40.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.234+0000 7ff140cee640 10 monclient(hunting): _negotiate 2024-03-25T14:29:40.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.234+0000 7ff140cee640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:40.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.234+0000 7ff140cee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:40.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.234+0000 7ff140cee640 10 monclient(hunting): my global_id is 4341 2024-03-25T14:29:40.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.234+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c0b28a0 con 0x7ff13c1430c0 2024-03-25T14:29:40.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.236+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3239817334 0 0) 0x7ff128003520 con 0x7ff13c0a98f0 2024-03-25T14:29:40.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.236+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 >> v1:172.21.15.26:6790/0 conn(0x7ff13c13f710 legacy=0x7ff13c1429b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T14:29:40.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 >> v1:172.21.15.26:6789/0 conn(0x7ff13c1430c0 legacy=0x7ff13c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: _finish_hunting 0 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 1 monclient: found mon.noname-c 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0b44c0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: _finish_auth 0 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: _check_auth_tickets 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff138001690 con 0x7ff13c0a98f0 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff140cee640 10 monclient: dump: 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:40.242 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff143f54640 5 monclient: authenticate success, global_id 4327 2024-03-25T14:29:40.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.237+0000 7ff143f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1380017f0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff140cee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff140cee640 1 -- 172.21.15.26:0/1787337469 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff138001c00 con 0x7ff13c0a98f0 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff143f54640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff143f54640 4 set_mon_vals no callback set 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff140cee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff140cee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.251+0000 7ff140cee640 10 monclient: dump: 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:40.256 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:40.257 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:40.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 10 monclient: shutdown 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 1 -- 172.21.15.26:0/1787337469 >> v1:172.21.15.121:6789/0 conn(0x7ff13c0a98f0 legacy=0x7ff13c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 1 -- 172.21.15.26:0/1787337469 shutdown_connections 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 1 -- 172.21.15.26:0/1787337469 >> 172.21.15.26:0/1787337469 conn(0x7ff13c0b7ea0 msgr2=0x7ff13c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 1 -- 172.21.15.26:0/1787337469 shutdown_connections 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.252+0000 7ff143f54640 1 -- 172.21.15.26:0/1787337469 wait complete. 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.253+0000 7ff143f54640 10 monclient: build_initial_monmap 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.253+0000 7ff143f54640 10 monclient: monmap: 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:40.260 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 librados: starting msgr at 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 librados: starting objecter 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 Processor -- start 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 -- start start 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 librados: setting wanted keys 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 librados: calling monclient init 2024-03-25T14:29:40.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 10 monclient: init 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 10 monclient(hunting): picked mon.noname-b con 0x7ff13c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 10 monclient(hunting): picked mon.noname-a con 0x7ff13c13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 10 monclient(hunting): picked mon.noname-c con 0x7ff13c1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c139510 con 0x7ff13c13f710 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.254+0000 7ff143f54640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c1366a0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.255+0000 7ff143f54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c1d7e80 con 0x7ff13c1430c0 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.255+0000 7ff143f54640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:40.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.255+0000 7ff143f54640 10 monclient(hunting): authenticate will time out at 1665.357613s 2024-03-25T14:29:40.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.261+0000 7ff1414ef640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff13c13f710 0x7ff13c0b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43050/0 (socket says 172.21.15.26:43050) 2024-03-25T14:29:40.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.261+0000 7ff1414ef640 1 -- 172.21.15.26:0/761747397 learned_addr learned my addr 172.21.15.26:0/761747397 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:40.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.264+0000 7ff1327fc640 10 client.?.objecter ms_handle_connect 0x7ff13c13f710 2024-03-25T14:29:40.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.264+0000 7ff1327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:40.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.268+0000 7ff1327fc640 10 client.?.objecter ms_handle_connect 0x7ff13c0a98f0 2024-03-25T14:29:40.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.268+0000 7ff1327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:40.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.273+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43043855 0 0) 0x7ff13c1366a0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.273+0000 7ff1327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:40.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.273+0000 7ff1327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:40.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.273+0000 7ff1327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:40.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.273+0000 7ff1327fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T14:29:40.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.273+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff120004eb0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.279+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886857476 0 0) 0x7ff13c139510 con 0x7ff13c13f710 2024-03-25T14:29:40.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.279+0000 7ff1327fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:40.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.279+0000 7ff1327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:40.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.279+0000 7ff1327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:40.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.279+0000 7ff1327fc640 10 monclient(hunting): my global_id is 4344 2024-03-25T14:29:40.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.279+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c1366a0 con 0x7ff13c13f710 2024-03-25T14:29:40.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 535690406 0 0) 0x7ff120004eb0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff13c139510 con 0x7ff13c0a98f0 2024-03-25T14:29:40.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff138001bc0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:40.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 10 monclient(hunting): dump: 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:40.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.280+0000 7ff1327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2593231333 0 0) 0x7ff13c139510 con 0x7ff13c0a98f0 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 >> v1:172.21.15.26:6790/0 conn(0x7ff13c1430c0 legacy=0x7ff13c1372d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 >> v1:172.21.15.26:6789/0 conn(0x7ff13c13f710 legacy=0x7ff13c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 10 monclient: _finish_hunting 0 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 1 monclient: found mon.b 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:40.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c1d8010 con 0x7ff13c0a98f0 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 10 monclient: _finish_auth 0 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 10 monclient: _check_auth_tickets 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff1327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 5 monclient: authenticate success, global_id 4330 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 10 monclient: _renew_subs 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c14e5e0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 10 monclient: _renew_subs 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c0b14b0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.287+0000 7ff143f54640 1 librados: init done 2024-03-25T14:29:40.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.289+0000 7ff143f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:40.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.289+0000 7ff143f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:40.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.289+0000 7ff143f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:40.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.289+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c1d8440 con 0x7ff13c0a98f0 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff138004a90 con 0x7ff13c0a98f0 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff1327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff138001250 con 0x7ff13c0a98f0 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff1327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff1327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff1327fc640 10 monclient: dump: 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:40.564 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:40.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.559+0000 7ff133fff640 4 set_mon_vals no callback set 2024-03-25T14:29:40.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.560+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ff13801acd0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.561+0000 7ff1414ef640 10 monclient: get_auth_request con 0x7ff12006dcb0 auth_method 0 2024-03-25T14:29:40.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.561+0000 7ff1327fc640 10 client.4330.objecter ms_handle_connect 0x7ff12006dcb0 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7ff1380548e0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 10 client.4330.objecter ms_dispatch 0x7ff13c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff138050220 con 0x7ff13c0a98f0 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:40.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.567+0000 7ff1327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:40.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.832+0000 7ff143f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:40.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.832+0000 7ff143f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:40.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.832+0000 7ff143f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:40.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.832+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff13c0b17b0 con 0x7ff13c0a98f0 2024-03-25T14:29:40.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.935+0000 7ff1327fc640 1 -- 172.21.15.26:0/761747397 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7ff138052320 con 0x7ff13c0a98f0 2024-03-25T14:29:40.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.935+0000 7ff1327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:40.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.935+0000 7ff1327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:40.940 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:40.940 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 10 librados: watch_flush enter 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 10 librados: watch_flush exit 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 10 client.4330.objecter successfully canceled tick 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 >> v1:172.21.15.121:6800/288813066 conn(0x7ff12006dcb0 legacy=0x7ff120070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 10 monclient: shutdown 2024-03-25T14:29:40.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:40.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 >> v1:172.21.15.121:6789/0 conn(0x7ff13c0a98f0 legacy=0x7ff13c1d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 shutdown_connections 2024-03-25T14:29:40.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 >> 172.21.15.26:0/761747397 conn(0x7ff13c0b7ea0 msgr2=0x7ff13c13d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:40.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 shutdown_connections 2024-03-25T14:29:40.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 -- 172.21.15.26:0/761747397 wait complete. 2024-03-25T14:29:40.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:40.937+0000 7ff143f54640 1 librados: shutdown 2024-03-25T14:29:40.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:47.761 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:47.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.865+0000 7f9d56d32640 10 monclient: get_monmap_and_config 2024-03-25T14:29:47.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.865+0000 7f9d56d32640 10 monclient: build_initial_monmap 2024-03-25T14:29:47.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.865+0000 7f9d56d32640 10 monclient: monmap: 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:47.865961+0000 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:47.865961+0000 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:47.870 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:47.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.865+0000 7f9d56d32640 1 Processor -- start 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.865+0000 7f9d56d32640 1 -- start start 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient: init 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient(hunting): picked mon.noname-c con 0x7f9d500a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient(hunting): picked mon.noname-a con 0x7f9d5013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient(hunting): picked mon.noname-b con 0x7f9d501430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d50142520 con 0x7f9d5013f710 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500b28a0 con 0x7f9d500a98f0 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500b2a30 con 0x7f9d501430c0 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:47.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.866+0000 7f9d56d32640 10 monclient(hunting): authenticate will time out at 1672.969275s 2024-03-25T14:29:47.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.873+0000 7f9d54ace640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9d500a98f0 0x7f9d500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39462/0 (socket says 172.21.15.26:39462) 2024-03-25T14:29:47.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.873+0000 7f9d54ace640 1 -- 172.21.15.26:0/3443039014 learned_addr learned my addr 172.21.15.26:0/3443039014 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:47.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.893+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637371218 0 0) 0x7f9d50142520 con 0x7f9d5013f710 2024-03-25T14:29:47.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.893+0000 7f9d4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.893+0000 7f9d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.893+0000 7f9d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.893+0000 7f9d4f7fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T14:29:47.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.893+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d40003520 con 0x7f9d5013f710 2024-03-25T14:29:47.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.895+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649313420 0 0) 0x7f9d500b28a0 con 0x7f9d500a98f0 2024-03-25T14:29:47.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.895+0000 7f9d4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.895+0000 7f9d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.895+0000 7f9d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.895+0000 7f9d4f7fe640 10 monclient(hunting): my global_id is 4333 2024-03-25T14:29:47.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.895+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50142520 con 0x7f9d500a98f0 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2671537326 0 0) 0x7f9d50142520 con 0x7f9d500a98f0 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 >> v1:172.21.15.26:6790/0 conn(0x7f9d501430c0 legacy=0x7f9d5014dc30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 >> v1:172.21.15.26:6789/0 conn(0x7f9d5013f710 legacy=0x7f9d501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 1 monclient: found mon.noname-c 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500b44c0 con 0x7f9d500a98f0 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:29:47.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:29:47.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:47.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9d44001670 con 0x7f9d500a98f0 2024-03-25T14:29:47.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:47.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d4f7fe640 10 monclient: dump: 2024-03-25T14:29:47.912 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d56d32640 5 monclient: authenticate success, global_id 4333 2024-03-25T14:29:47.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.906+0000 7f9d56d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:47.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d440017d0 con 0x7f9d500a98f0 2024-03-25T14:29:47.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:47.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d4f7fe640 1 -- 172.21.15.26:0/3443039014 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9d44001be0 con 0x7f9d500a98f0 2024-03-25T14:29:47.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d56d32640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:47.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d56d32640 4 set_mon_vals no callback set 2024-03-25T14:29:47.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:47.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.922+0000 7f9d4f7fe640 10 monclient: dump: 2024-03-25T14:29:47.930 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:47.930 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:47.931 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 10 monclient: shutdown 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 1 -- 172.21.15.26:0/3443039014 >> v1:172.21.15.121:6789/0 conn(0x7f9d500a98f0 legacy=0x7f9d500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 1 -- 172.21.15.26:0/3443039014 shutdown_connections 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 1 -- 172.21.15.26:0/3443039014 >> 172.21.15.26:0/3443039014 conn(0x7f9d500b7ea0 msgr2=0x7f9d500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:47.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 1 -- 172.21.15.26:0/3443039014 shutdown_connections 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.923+0000 7f9d56d32640 1 -- 172.21.15.26:0/3443039014 wait complete. 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 10 monclient: build_initial_monmap 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 10 monclient: monmap: 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:47.933 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:47.934 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:47.934 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:47.934 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:47.934 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:47.934 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:47.934 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:47.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 1 librados: starting msgr at 2024-03-25T14:29:47.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 1 librados: starting objecter 2024-03-25T14:29:47.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 1 Processor -- start 2024-03-25T14:29:47.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 1 -- start start 2024-03-25T14:29:47.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 1 librados: setting wanted keys 2024-03-25T14:29:47.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 1 librados: calling monclient init 2024-03-25T14:29:47.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.924+0000 7f9d56d32640 10 monclient: init 2024-03-25T14:29:47.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:47.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:29:47.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient(hunting): picked mon.noname-a con 0x7f9d500a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient(hunting): picked mon.noname-b con 0x7f9d5013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient(hunting): picked mon.noname-c con 0x7f9d501430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d501dac70 con 0x7f9d500a98f0 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d500ab950 con 0x7f9d5013f710 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d501db810 con 0x7f9d501430c0 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.925+0000 7f9d56d32640 10 monclient(hunting): authenticate will time out at 1673.028369s 2024-03-25T14:29:47.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.928+0000 7f9d4ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9d5013f710 0x7f9d501db100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39524/0 (socket says 172.21.15.26:39524) 2024-03-25T14:29:47.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.928+0000 7f9d4ffff640 1 -- 172.21.15.26:0/2655843610 learned_addr learned my addr 172.21.15.26:0/2655843610 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:47.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.930+0000 7f9d4d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d5013f710 2024-03-25T14:29:47.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.930+0000 7f9d4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:47.941 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.937+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863722965 0 0) 0x7f9d500ab950 con 0x7f9d5013f710 2024-03-25T14:29:47.941 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.937+0000 7f9d4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.937+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.937+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.937+0000 7f9d4d7fa640 10 monclient(hunting): my global_id is 4336 2024-03-25T14:29:47.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.937+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34004eb0 con 0x7f9d5013f710 2024-03-25T14:29:47.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.944+0000 7f9d4d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d500a98f0 2024-03-25T14:29:47.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.944+0000 7f9d4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:47.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.947+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361313548 0 0) 0x7f9d34004eb0 con 0x7f9d5013f710 2024-03-25T14:29:47.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.947+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d500ab950 con 0x7f9d5013f710 2024-03-25T14:29:47.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.947+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9d3c001a10 con 0x7f9d5013f710 2024-03-25T14:29:47.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.947+0000 7f9d4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.948+0000 7f9d4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.948+0000 7f9d4d7fa640 10 monclient(hunting): dump: 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:47.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.948+0000 7f9d4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:47.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.951+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477031045 0 0) 0x7f9d501dac70 con 0x7f9d500a98f0 2024-03-25T14:29:47.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T14:29:47.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34004eb0 con 0x7f9d500a98f0 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1263778850 0 0) 0x7f9d500ab950 con 0x7f9d5013f710 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 >> v1:172.21.15.26:6790/0 conn(0x7f9d501430c0 legacy=0x7f9d50134010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 >> v1:172.21.15.26:6789/0 conn(0x7f9d500a98f0 legacy=0x7f9d50137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 1 monclient: found mon.b 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:47.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d501db9a0 con 0x7f9d5013f710 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 5 monclient: authenticate success, global_id 4336 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 10 monclient: _renew_subs 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d50134de0 con 0x7f9d5013f710 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 10 monclient: _renew_subs 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d50134900 con 0x7f9d5013f710 2024-03-25T14:29:47.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.952+0000 7f9d56d32640 1 librados: init done 2024-03-25T14:29:47.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.953+0000 7f9d56d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:47.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.953+0000 7f9d56d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:47.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.953+0000 7f9d56d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:47.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.953+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d500627b0 con 0x7f9d5013f710 2024-03-25T14:29:47.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d3c001cd0 con 0x7f9d5013f710 2024-03-25T14:29:47.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:47.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9d3c003a60 con 0x7f9d5013f710 2024-03-25T14:29:47.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:47.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4d7fa640 10 monclient: dump: 2024-03-25T14:29:47.980 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:47.980 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:47.981 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.981 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:47.981 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.981 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:47.981 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:47.981 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:47.982 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:47.982 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:47.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.974+0000 7f9d4effd640 4 set_mon_vals no callback set 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.975+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f9d3c003c80 con 0x7f9d5013f710 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.976+0000 7f9d54ace640 10 monclient: get_auth_request con 0x7f9d3406dcc0 auth_method 0 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:47.976+0000 7f9d4d7fa640 10 client.4336.objecter ms_handle_connect 0x7f9d3406dcc0 2024-03-25T14:29:48.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.221+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f9d3c0548f0 con 0x7f9d5013f710 2024-03-25T14:29:48.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.221+0000 7f9d4d7fa640 10 client.4336.objecter ms_dispatch 0x7f9d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:48.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.221+0000 7f9d4d7fa640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:48.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.221+0000 7f9d4d7fa640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:48.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.221+0000 7f9d4d7fa640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T14:29:48.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.225+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d3c050400 con 0x7f9d5013f710 2024-03-25T14:29:48.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.225+0000 7f9d4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:48.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.225+0000 7f9d4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:48.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.445+0000 7f9d56d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:48.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.445+0000 7f9d56d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:48.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.445+0000 7f9d56d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:48.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.445+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d50141a00 con 0x7f9d5013f710 2024-03-25T14:29:48.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.463+0000 7f9d4d7fa640 1 -- 172.21.15.26:0/2655843610 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f9d3c052330 con 0x7f9d5013f710 2024-03-25T14:29:48.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.463+0000 7f9d4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:48.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.463+0000 7f9d4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:48.468 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:48.468 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 10 librados: watch_flush enter 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 10 librados: watch_flush exit 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 10 client.4336.objecter successfully canceled tick 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 >> v1:172.21.15.121:6800/288813066 conn(0x7f9d3406dcc0 legacy=0x7f9d34070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 10 monclient: shutdown 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 >> v1:172.21.15.121:6789/0 conn(0x7f9d5013f710 legacy=0x7f9d501db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 shutdown_connections 2024-03-25T14:29:48.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 >> 172.21.15.26:0/2655843610 conn(0x7f9d500b7ea0 msgr2=0x7f9d50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:48.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 shutdown_connections 2024-03-25T14:29:48.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 -- 172.21.15.26:0/2655843610 wait complete. 2024-03-25T14:29:48.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:48.465+0000 7f9d56d32640 1 librados: shutdown 2024-03-25T14:29:48.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:50.855 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:09.716 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:50.904 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:09.765 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:50.945 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:09.806 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:50.995 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:09.871 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:51.007 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:09.884 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:51.041 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:09.918 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:54.719 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:13.580 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:54.725 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:13.586 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:54.759 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:13.620 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:55.288 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.402+0000 7f219d115640 10 monclient: get_monmap_and_config 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.402+0000 7f219d115640 10 monclient: build_initial_monmap 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.402+0000 7f219d115640 10 monclient: monmap: 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:29:55.403276+0000 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:29:55.403276+0000 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:29:55.407 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:29:55.408 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 1 Processor -- start 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 1 -- start start 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient: init 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient(hunting): picked mon.noname-a con 0x7f21980a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient(hunting): picked mon.noname-b con 0x7f219813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:55.409 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient(hunting): picked mon.noname-c con 0x7f21981430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:55.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2198142520 con 0x7f21980a98f0 2024-03-25T14:29:55.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21980b28a0 con 0x7f21981430c0 2024-03-25T14:29:55.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21980b2a30 con 0x7f219813f710 2024-03-25T14:29:55.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:55.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.403+0000 7f219d115640 10 monclient(hunting): authenticate will time out at 1680.506446s 2024-03-25T14:29:55.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.414+0000 7f218e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f219813f710 0x7f21981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50892/0 (socket says 172.21.15.26:50892) 2024-03-25T14:29:55.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.414+0000 7f218e59c640 1 -- 172.21.15.26:0/2003302415 learned_addr learned my addr 172.21.15.26:0/2003302415 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:55.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.430+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234045183 0 0) 0x7f21980b28a0 con 0x7f21981430c0 2024-03-25T14:29:55.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.430+0000 7f219659c640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.430+0000 7f219659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.430+0000 7f219659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.430+0000 7f219659c640 10 monclient(hunting): my global_id is 4339 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.430+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2184003520 con 0x7f21981430c0 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766299591 0 0) 0x7f21980b2a30 con 0x7f219813f710 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): my global_id is 4313 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21980b28a0 con 0x7f219813f710 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017221152 0 0) 0x7f2198142520 con 0x7f21980a98f0 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 10 monclient(hunting): my global_id is 4353 2024-03-25T14:29:55.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.431+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21980b2a30 con 0x7f21980a98f0 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893299334 0 0) 0x7f2184003520 con 0x7f21981430c0 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 >> v1:172.21.15.26:6790/0 conn(0x7f219813f710 legacy=0x7f21981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 >> v1:172.21.15.26:6789/0 conn(0x7f21980a98f0 legacy=0x7f21980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: _finish_hunting 0 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 1 monclient: found mon.noname-c 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21980b44c0 con 0x7f21981430c0 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: _finish_auth 0 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: _check_auth_tickets 2024-03-25T14:29:55.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2190001a10 con 0x7f21981430c0 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219659c640 10 monclient: dump: 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:55.447 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219d115640 5 monclient: authenticate success, global_id 4339 2024-03-25T14:29:55.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.441+0000 7f219d115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2190001b70 con 0x7f21981430c0 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219659c640 1 -- 172.21.15.26:0/2003302415 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2190001f80 con 0x7f21981430c0 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219d115640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219d115640 4 set_mon_vals no callback set 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.468+0000 7f219659c640 10 monclient: dump: 2024-03-25T14:29:55.473 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:55.474 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 10 monclient: shutdown 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 1 -- 172.21.15.26:0/2003302415 >> v1:172.21.15.121:6789/0 conn(0x7f21981430c0 legacy=0x7f219814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 1 -- 172.21.15.26:0/2003302415 shutdown_connections 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 1 -- 172.21.15.26:0/2003302415 >> 172.21.15.26:0/2003302415 conn(0x7f21980b7ea0 msgr2=0x7f21980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 1 -- 172.21.15.26:0/2003302415 shutdown_connections 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.469+0000 7f219d115640 1 -- 172.21.15.26:0/2003302415 wait complete. 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.470+0000 7f219d115640 10 monclient: build_initial_monmap 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.470+0000 7f219d115640 10 monclient: monmap: 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:29:55.477 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.470+0000 7f219d115640 1 librados: starting msgr at 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.470+0000 7f219d115640 1 librados: starting objecter 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.470+0000 7f219d115640 1 Processor -- start 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 1 -- start start 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 1 librados: setting wanted keys 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 1 librados: calling monclient init 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 10 monclient: init 2024-03-25T14:29:55.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 10 monclient(hunting): picked mon.noname-a con 0x7f21980a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 10 monclient(hunting): picked mon.noname-b con 0x7f219813f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 10 monclient(hunting): picked mon.noname-c con 0x7f21981430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2198138a70 con 0x7f21980a98f0 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.471+0000 7f219d115640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21980aba50 con 0x7f219813f710 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.472+0000 7f219d115640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21981d77a0 con 0x7f21981430c0 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.472+0000 7f219d115640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:55.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.472+0000 7f219d115640 10 monclient(hunting): authenticate will time out at 1680.574621s 2024-03-25T14:29:55.488 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.484+0000 7f2196d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f21980a98f0 0x7f2198138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43106/0 (socket says 172.21.15.26:43106) 2024-03-25T14:29:55.488 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.484+0000 7f2196d9d640 1 -- 172.21.15.26:0/3741722342 learned_addr learned my addr 172.21.15.26:0/3741722342 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:29:55.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.487+0000 7f218ffff640 10 client.?.objecter ms_handle_connect 0x7f219813f710 2024-03-25T14:29:55.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.487+0000 7f218ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:55.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.488+0000 7f218ffff640 10 client.?.objecter ms_handle_connect 0x7f21980a98f0 2024-03-25T14:29:55.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.488+0000 7f218ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:55.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.488+0000 7f218ffff640 10 client.?.objecter ms_handle_connect 0x7f21981430c0 2024-03-25T14:29:55.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.488+0000 7f218ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:55.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.497+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325340772 0 0) 0x7f2198138a70 con 0x7f21980a98f0 2024-03-25T14:29:55.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.497+0000 7f218ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.497+0000 7f218ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.497+0000 7f218ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.497+0000 7f218ffff640 10 monclient(hunting): my global_id is 4356 2024-03-25T14:29:55.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.497+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2178004eb0 con 0x7f21980a98f0 2024-03-25T14:29:55.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.501+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713793792 0 0) 0x7f21981d77a0 con 0x7f21981430c0 2024-03-25T14:29:55.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.501+0000 7f218ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.501+0000 7f218ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.501+0000 7f218ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.501+0000 7f218ffff640 10 monclient(hunting): my global_id is 4316 2024-03-25T14:29:55.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.501+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198138a70 con 0x7f21981430c0 2024-03-25T14:29:55.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.507+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767064821 0 0) 0x7f21980aba50 con 0x7f219813f710 2024-03-25T14:29:55.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.507+0000 7f218ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.507+0000 7f218ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.507+0000 7f218ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.507+0000 7f218ffff640 10 monclient(hunting): my global_id is 4342 2024-03-25T14:29:55.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.507+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21981d77a0 con 0x7f219813f710 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2006346020 0 0) 0x7f2178004eb0 con 0x7f21980a98f0 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21980aba50 con 0x7f21980a98f0 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2188001a30 con 0x7f21980a98f0 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 10 monclient(hunting): dump: 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.522 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.523 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.517+0000 7f218ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.527+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2337537827 0 0) 0x7f21981d77a0 con 0x7f219813f710 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.527+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2178004eb0 con 0x7f219813f710 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.527+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2180001930 con 0x7f219813f710 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.527+0000 7f218ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.527+0000 7f218ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.527+0000 7f218ffff640 10 monclient(hunting): dump: 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.532 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.533 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:55.533 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:55.533 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:55.533 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.536+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436483210 0 0) 0x7f2198138a70 con 0x7f21981430c0 2024-03-25T14:29:55.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.536+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21981d77a0 con 0x7f21981430c0 2024-03-25T14:29:55.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.536+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2190001e80 con 0x7f21981430c0 2024-03-25T14:29:55.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.536+0000 7f218ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.536+0000 7f218ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.536+0000 7f218ffff640 10 monclient(hunting): dump: 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:55.541 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.539+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1531263104 0 0) 0x7f2178004eb0 con 0x7f219813f710 2024-03-25T14:29:55.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.539+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 >> v1:172.21.15.26:6790/0 conn(0x7f21981430c0 legacy=0x7f21981d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.539+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 >> v1:172.21.15.26:6789/0 conn(0x7f21980a98f0 legacy=0x7f2198138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.539+0000 7f218ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.539+0000 7f218ffff640 1 monclient: found mon.b 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.539+0000 7f218ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f218ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21981d83d0 con 0x7f219813f710 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f218ffff640 10 monclient: _finish_auth 0 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f218ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f218ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 5 monclient: authenticate success, global_id 4342 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 10 monclient: _renew_subs 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21981d41e0 con 0x7f219813f710 2024-03-25T14:29:55.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:55.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 10 monclient: _renew_subs 2024-03-25T14:29:55.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:55.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21981d5110 con 0x7f219813f710 2024-03-25T14:29:55.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 1 librados: init done 2024-03-25T14:29:55.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.540+0000 7f219d115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:55.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.541+0000 7f219d115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:55.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.541+0000 7f219d115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:55.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.541+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219814f320 con 0x7f219813f710 2024-03-25T14:29:55.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21800047e0 con 0x7f219813f710 2024-03-25T14:29:55.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f218ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:55.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2180004af0 con 0x7f219813f710 2024-03-25T14:29:55.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f218ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f218ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f218ffff640 10 monclient: dump: 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:29:55.858 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:29:55.859 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:29:55.859 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:29:55.859 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:29:55.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.852+0000 7f2195d9b640 4 set_mon_vals no callback set 2024-03-25T14:29:55.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.854+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f218001aac0 con 0x7f219813f710 2024-03-25T14:29:55.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.854+0000 7f2196d9d640 10 monclient: get_auth_request con 0x7f217806e160 auth_method 0 2024-03-25T14:29:55.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.854+0000 7f218ffff640 10 client.4342.objecter ms_handle_connect 0x7f217806e160 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f2180054720 con 0x7f219813f710 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 10 client.4342.objecter ms_dispatch 0x7f2198000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2180050230 con 0x7f219813f710 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:55.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:55.859+0000 7f218ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:56.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.118+0000 7f219d115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:56.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.118+0000 7f219d115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:56.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.118+0000 7f219d115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:29:56.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.118+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21981d7930 con 0x7f219813f710 2024-03-25T14:29:56.150 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.145+0000 7f218ffff640 1 -- 172.21.15.26:0/3741722342 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f2180052160 con 0x7f219813f710 2024-03-25T14:29:56.150 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.145+0000 7f218ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:56.150 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.145+0000 7f218ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:56.150 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:29:56.151 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:56.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 10 librados: watch_flush enter 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 10 librados: watch_flush exit 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 10 client.4342.objecter successfully canceled tick 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 >> v1:172.21.15.121:6800/288813066 conn(0x7f217806e160 legacy=0x7f2178070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 10 monclient: shutdown 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 >> v1:172.21.15.121:6789/0 conn(0x7f219813f710 legacy=0x7f21981d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 shutdown_connections 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.148+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 >> 172.21.15.26:0/3741722342 conn(0x7f21980b7ea0 msgr2=0x7f219813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.149+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 shutdown_connections 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.149+0000 7f219d115640 1 -- 172.21.15.26:0/3741722342 wait complete. 2024-03-25T14:29:56.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:29:56.149+0000 7f219d115640 1 librados: shutdown 2024-03-25T14:29:56.166 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:02.973 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:03.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.083+0000 7fa56700d640 10 monclient: get_monmap_and_config 2024-03-25T14:30:03.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.083+0000 7fa56700d640 10 monclient: build_initial_monmap 2024-03-25T14:30:03.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.083+0000 7fa56700d640 10 monclient: monmap: 2024-03-25T14:30:03.087 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:03.087 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:03.083948+0000 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:03.083948+0000 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:03.088 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 1 Processor -- start 2024-03-25T14:30:03.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 1 -- start start 2024-03-25T14:30:03.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 10 monclient: init 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 10 monclient(hunting): picked mon.noname-c con 0x7fa56006f3f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 10 monclient(hunting): picked mon.noname-a con 0x7fa56006f7d0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5600726a0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.084+0000 7fa56700d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa560071440 con 0x7fa56006f7d0 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.085+0000 7fa56700d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5600716b0 con 0x7fa56006f3f0 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.085+0000 7fa56700d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5600ae960 con 0x7fa5600726a0 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.085+0000 7fa56700d640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:03.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.085+0000 7fa56700d640 10 monclient(hunting): authenticate will time out at 1688.187700s 2024-03-25T14:30:03.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.097+0000 7fa5655aa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa5600726a0 0x7fa56014a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56940/0 (socket says 172.21.15.26:56940) 2024-03-25T14:30:03.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.097+0000 7fa5655aa640 1 -- 172.21.15.26:0/2304014501 learned_addr learned my addr 172.21.15.26:0/2304014501 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:03.112 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120508354 0 0) 0x7fa5600ae960 con 0x7fa5600726a0 2024-03-25T14:30:03.112 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.112 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.112 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.112 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 10 monclient(hunting): my global_id is 4319 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa548003520 con 0x7fa5600726a0 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.107+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196464783 0 0) 0x7fa5600716b0 con 0x7fa56006f3f0 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.108+0000 7fa5577fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.108+0000 7fa5577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.108+0000 7fa5577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.108+0000 7fa5577fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T14:30:03.113 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.108+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5600ae960 con 0x7fa56006f3f0 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409668644 0 0) 0x7fa5600ae960 con 0x7fa56006f3f0 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 >> v1:172.21.15.26:6790/0 conn(0x7fa5600726a0 legacy=0x7fa56014a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 >> v1:172.21.15.26:6789/0 conn(0x7fa56006f7d0 legacy=0x7fa560071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: _finish_hunting 0 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 1 monclient: found mon.noname-c 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:30:03.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560072ab0 con 0x7fa56006f3f0 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: _finish_auth 0 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: _check_auth_tickets 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa55c001670 con 0x7fa56006f3f0 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa5577fe640 10 monclient: dump: 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:03.122 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:03.123 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:03.123 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa56700d640 5 monclient: authenticate success, global_id 4345 2024-03-25T14:30:03.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.116+0000 7fa56700d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa55c0017d0 con 0x7fa56006f3f0 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa5577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa5577fe640 1 -- 172.21.15.26:0/2304014501 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa55c001c20 con 0x7fa56006f3f0 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa56700d640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa56700d640 4 set_mon_vals no callback set 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa5577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa5577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.125+0000 7fa5577fe640 10 monclient: dump: 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.130 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.131 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.131 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.131 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:03.131 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:03.131 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:03.131 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 10 monclient: shutdown 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 1 -- 172.21.15.26:0/2304014501 >> v1:172.21.15.121:6789/0 conn(0x7fa56006f3f0 legacy=0x7fa56013c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 1 -- 172.21.15.26:0/2304014501 shutdown_connections 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 1 -- 172.21.15.26:0/2304014501 >> 172.21.15.26:0/2304014501 conn(0x7fa5600b4bf0 msgr2=0x7fa5600b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 1 -- 172.21.15.26:0/2304014501 shutdown_connections 2024-03-25T14:30:03.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.126+0000 7fa56700d640 1 -- 172.21.15.26:0/2304014501 wait complete. 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 10 monclient: build_initial_monmap 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 10 monclient: monmap: 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 1 librados: starting msgr at 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 1 librados: starting objecter 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 1 Processor -- start 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 1 -- start start 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 1 librados: setting wanted keys 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 1 librados: calling monclient init 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.127+0000 7fa56700d640 10 monclient: init 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient(hunting): picked mon.noname-b con 0x7fa56006f3f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient(hunting): picked mon.noname-a con 0x7fa56006f7d0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5600726a0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa560130720 con 0x7fa56006f7d0 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5600b3ef0 con 0x7fa56006f3f0 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa560137160 con 0x7fa5600726a0 2024-03-25T14:30:03.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:03.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.128+0000 7fa56700d640 10 monclient(hunting): authenticate will time out at 1688.231469s 2024-03-25T14:30:03.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.136+0000 7fa5655aa640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa5600726a0 0x7fa560130010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56960/0 (socket says 172.21.15.26:56960) 2024-03-25T14:30:03.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.136+0000 7fa5655aa640 1 -- 172.21.15.26:0/3132629273 learned_addr learned my addr 172.21.15.26:0/3132629273 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:03.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.137+0000 7fa5557fa640 10 client.?.objecter ms_handle_connect 0x7fa5600726a0 2024-03-25T14:30:03.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.137+0000 7fa5557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:03.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 10 client.?.objecter ms_handle_connect 0x7fa56006f7d0 2024-03-25T14:30:03.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:03.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960073643 0 0) 0x7fa560137160 con 0x7fa5600726a0 2024-03-25T14:30:03.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 10 monclient(hunting): my global_id is 4322 2024-03-25T14:30:03.153 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.147+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa54c004eb0 con 0x7fa5600726a0 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087842682 0 0) 0x7fa560130720 con 0x7fa56006f7d0 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 10 monclient(hunting): my global_id is 4359 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560137160 con 0x7fa56006f7d0 2024-03-25T14:30:03.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 10 client.?.objecter ms_handle_connect 0x7fa56006f3f0 2024-03-25T14:30:03.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.155+0000 7fa5557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:03.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.163+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174746788 0 0) 0x7fa5600b3ef0 con 0x7fa56006f3f0 2024-03-25T14:30:03.167 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.163+0000 7fa5557fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.163+0000 7fa5557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.163+0000 7fa5557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.163+0000 7fa5557fa640 10 monclient(hunting): my global_id is 4348 2024-03-25T14:30:03.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.163+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560130720 con 0x7fa56006f3f0 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1228142695 0 0) 0x7fa54c004eb0 con 0x7fa5600726a0 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5600b3ef0 con 0x7fa5600726a0 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa558001c10 con 0x7fa5600726a0 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 10 monclient(hunting): dump: 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.170 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.171 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.165+0000 7fa5557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.168+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3363086971 0 0) 0x7fa560137160 con 0x7fa56006f7d0 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.168+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa54c004eb0 con 0x7fa56006f7d0 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.168+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa550001830 con 0x7fa56006f7d0 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.168+0000 7fa5557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.168+0000 7fa5557fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.168+0000 7fa5557fa640 10 monclient(hunting): dump: 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.173 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.174 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:03.174 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:03.174 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:03.174 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.172+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567722631 0 0) 0x7fa560130720 con 0x7fa56006f3f0 2024-03-25T14:30:03.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.172+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa560137160 con 0x7fa56006f3f0 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.172+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa55c001ae0 con 0x7fa56006f3f0 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.172+0000 7fa5557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.172+0000 7fa5557fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.172+0000 7fa5557fa640 10 monclient(hunting): dump: 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.177 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.178 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.178 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:03.178 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:03.179 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:03.179 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3682038177 0 0) 0x7fa560137160 con 0x7fa56006f3f0 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 >> v1:172.21.15.26:6790/0 conn(0x7fa5600726a0 legacy=0x7fa560130010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 >> v1:172.21.15.26:6789/0 conn(0x7fa56006f7d0 legacy=0x7fa56012f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 10 monclient: _finish_hunting 0 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 1 monclient: found mon.b 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560137440 con 0x7fa56006f3f0 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 10 monclient: _finish_auth 0 2024-03-25T14:30:03.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 10 monclient: _check_auth_tickets 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa5557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 5 monclient: authenticate success, global_id 4348 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 10 monclient: _renew_subs 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa560137720 con 0x7fa56006f3f0 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 10 monclient: _renew_subs 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5600b4190 con 0x7fa56006f3f0 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.179+0000 7fa56700d640 1 librados: init done 2024-03-25T14:30:03.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.180+0000 7fa56700d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:03.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.180+0000 7fa56700d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:03.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.180+0000 7fa56700d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:03.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.181+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5600627b0 con 0x7fa56006f3f0 2024-03-25T14:30:03.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa55c001210 con 0x7fa56006f3f0 2024-03-25T14:30:03.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa55c0038a0 con 0x7fa56006f3f0 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 10 monclient: dump: 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:03.461 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:03.462 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:03.462 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:03.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa556ffd640 4 set_mon_vals no callback set 2024-03-25T14:30:03.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.456+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fa55c0198f0 con 0x7fa56006f3f0 2024-03-25T14:30:03.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.457+0000 7fa557fff640 10 monclient: get_auth_request con 0x7fa54c06e160 auth_method 0 2024-03-25T14:30:03.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.457+0000 7fa5557fa640 10 client.4348.objecter ms_handle_connect 0x7fa54c06e160 2024-03-25T14:30:03.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.463+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fa55c0535a0 con 0x7fa56006f3f0 2024-03-25T14:30:03.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.463+0000 7fa5557fa640 10 client.4348.objecter ms_dispatch 0x7fa560000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:03.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.463+0000 7fa5557fa640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:03.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.463+0000 7fa5557fa640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:03.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.464+0000 7fa5557fa640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T14:30:03.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.464+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa55c04eee0 con 0x7fa56006f3f0 2024-03-25T14:30:03.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.464+0000 7fa5557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:03.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.464+0000 7fa5557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:03.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.726+0000 7fa56700d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:03.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.726+0000 7fa56700d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:03.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.726+0000 7fa56700d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:03.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.726+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa56013ff30 con 0x7fa56006f3f0 2024-03-25T14:30:03.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.744+0000 7fa5557fa640 1 -- 172.21.15.26:0/3132629273 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fa55c050fe0 con 0x7fa56006f3f0 2024-03-25T14:30:03.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.744+0000 7fa5557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:03.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.744+0000 7fa5557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:03.749 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:03.749 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 10 librados: watch_flush enter 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 10 librados: watch_flush exit 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 10 client.4348.objecter successfully canceled tick 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 >> v1:172.21.15.121:6800/288813066 conn(0x7fa54c06e160 legacy=0x7fa54c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 10 monclient: shutdown 2024-03-25T14:30:03.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:03.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 >> v1:172.21.15.121:6789/0 conn(0x7fa56006f3f0 legacy=0x7fa560137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 shutdown_connections 2024-03-25T14:30:03.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 >> 172.21.15.26:0/3132629273 conn(0x7fa5600b4bf0 msgr2=0x7fa5600b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:03.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 shutdown_connections 2024-03-25T14:30:03.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 -- 172.21.15.26:0/3132629273 wait complete. 2024-03-25T14:30:03.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:03.746+0000 7fa56700d640 1 librados: shutdown 2024-03-25T14:30:03.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:10.571 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:10.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.691+0000 7fa1845cf640 10 monclient: get_monmap_and_config 2024-03-25T14:30:10.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.691+0000 7fa1845cf640 10 monclient: build_initial_monmap 2024-03-25T14:30:10.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.691+0000 7fa1845cf640 10 monclient: monmap: 2024-03-25T14:30:10.696 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:10.691957+0000 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:10.691957+0000 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.691+0000 7fa1845cf640 1 Processor -- start 2024-03-25T14:30:10.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.691+0000 7fa1845cf640 1 -- start start 2024-03-25T14:30:10.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.691+0000 7fa1845cf640 10 monclient: init 2024-03-25T14:30:10.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:10.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:30:10.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:10.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:10.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:10.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c142520 con 0x7fa17c0a98f0 2024-03-25T14:30:10.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0b28a0 con 0x7fa17c1430c0 2024-03-25T14:30:10.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0b2a30 con 0x7fa17c13f710 2024-03-25T14:30:10.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:10.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.692+0000 7fa1845cf640 10 monclient(hunting): authenticate will time out at 1695.795523s 2024-03-25T14:30:10.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.705+0000 7fa182b6c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa17c1430c0 0x7fa17c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:51886/0 (socket says 172.21.15.26:51886) 2024-03-25T14:30:10.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.705+0000 7fa182b6c640 1 -- 172.21.15.26:0/3088179144 learned_addr learned my addr 172.21.15.26:0/3088179144 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:10.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.714+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939240067 0 0) 0x7fa17c0b28a0 con 0x7fa17c1430c0 2024-03-25T14:30:10.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.714+0000 7fa181369640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.714+0000 7fa181369640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.714+0000 7fa181369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.714+0000 7fa181369640 10 monclient(hunting): my global_id is 4351 2024-03-25T14:30:10.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.714+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa168003520 con 0x7fa17c1430c0 2024-03-25T14:30:10.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1405671823 0 0) 0x7fa168003520 con 0x7fa17c1430c0 2024-03-25T14:30:10.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 >> v1:172.21.15.26:6790/0 conn(0x7fa17c13f710 legacy=0x7fa17c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 >> v1:172.21.15.26:6789/0 conn(0x7fa17c0a98f0 legacy=0x7fa17c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: _finish_hunting 0 2024-03-25T14:30:10.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 1 monclient: found mon.noname-c 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0b44c0 con 0x7fa17c1430c0 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: _finish_auth 0 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: _check_auth_tickets 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa174001a10 con 0x7fa17c1430c0 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa1845cf640 5 monclient: authenticate success, global_id 4351 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa1845cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:10.733 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:10.734 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa181369640 10 monclient: dump: 2024-03-25T14:30:10.734 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:10.734 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:10.734 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.734 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.735 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.737 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:10.737 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:10.738 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:10.738 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:10.738 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:10.738 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.725+0000 7fa1845cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:10.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa174001b70 con 0x7fa17c1430c0 2024-03-25T14:30:10.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa181369640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:10.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa181369640 1 -- 172.21.15.26:0/3088179144 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa174001f80 con 0x7fa17c1430c0 2024-03-25T14:30:10.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa1845cf640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:10.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa1845cf640 4 set_mon_vals no callback set 2024-03-25T14:30:10.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa181369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.743+0000 7fa181369640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:10.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.744+0000 7fa181369640 10 monclient: dump: 2024-03-25T14:30:10.749 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:10.749 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:10.749 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.749 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.750 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.750 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:10.750 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:10.750 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:10.750 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:10.750 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:10.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.744+0000 7fa1845cf640 10 monclient: shutdown 2024-03-25T14:30:10.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.744+0000 7fa1845cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.744+0000 7fa1845cf640 1 -- 172.21.15.26:0/3088179144 >> v1:172.21.15.121:6789/0 conn(0x7fa17c1430c0 legacy=0x7fa17c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.744+0000 7fa1845cf640 1 -- 172.21.15.26:0/3088179144 shutdown_connections 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.744+0000 7fa1845cf640 1 -- 172.21.15.26:0/3088179144 >> 172.21.15.26:0/3088179144 conn(0x7fa17c0b7ea0 msgr2=0x7fa17c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.745+0000 7fa1845cf640 1 -- 172.21.15.26:0/3088179144 shutdown_connections 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.745+0000 7fa1845cf640 1 -- 172.21.15.26:0/3088179144 wait complete. 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.745+0000 7fa1845cf640 10 monclient: build_initial_monmap 2024-03-25T14:30:10.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.745+0000 7fa1845cf640 10 monclient: monmap: 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:10.754 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 librados: starting msgr at 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 librados: starting objecter 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 Processor -- start 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 -- start start 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 librados: setting wanted keys 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 librados: calling monclient init 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient: init 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:10.755 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c136ad0 con 0x7fa17c1430c0 2024-03-25T14:30:10.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0b7520 con 0x7fa17c0a98f0 2024-03-25T14:30:10.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c135810 con 0x7fa17c13f710 2024-03-25T14:30:10.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:10.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.746+0000 7fa1845cf640 10 monclient(hunting): authenticate will time out at 1695.849547s 2024-03-25T14:30:10.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.759+0000 7fa18236b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa17c0a98f0 0x7fa17c135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:51922/0 (socket says 172.21.15.26:51922) 2024-03-25T14:30:10.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.759+0000 7fa18236b640 1 -- 172.21.15.26:0/1695547154 learned_addr learned my addr 172.21.15.26:0/1695547154 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:10.767 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.762+0000 7fa172ffd640 10 client.?.objecter ms_handle_connect 0x7fa17c0a98f0 2024-03-25T14:30:10.767 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.762+0000 7fa172ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:10.772 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.767+0000 7fa172ffd640 10 client.?.objecter ms_handle_connect 0x7fa17c1430c0 2024-03-25T14:30:10.772 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.767+0000 7fa172ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:10.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.769+0000 7fa172ffd640 10 client.?.objecter ms_handle_connect 0x7fa17c13f710 2024-03-25T14:30:10.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.769+0000 7fa172ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:10.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.776+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245250471 0 0) 0x7fa17c135810 con 0x7fa17c13f710 2024-03-25T14:30:10.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.776+0000 7fa172ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.776+0000 7fa172ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.776+0000 7fa172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.776+0000 7fa172ffd640 10 monclient(hunting): my global_id is 4328 2024-03-25T14:30:10.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.776+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa160004eb0 con 0x7fa17c13f710 2024-03-25T14:30:10.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.777+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012935516 0 0) 0x7fa17c0b7520 con 0x7fa17c0a98f0 2024-03-25T14:30:10.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.777+0000 7fa172ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.777+0000 7fa172ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.777+0000 7fa172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.777+0000 7fa172ffd640 10 monclient(hunting): my global_id is 4354 2024-03-25T14:30:10.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.777+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c135810 con 0x7fa17c0a98f0 2024-03-25T14:30:10.787 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.782+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497144073 0 0) 0x7fa17c136ad0 con 0x7fa17c1430c0 2024-03-25T14:30:10.787 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.782+0000 7fa172ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.787 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.782+0000 7fa172ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.787 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.782+0000 7fa172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.787 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.782+0000 7fa172ffd640 10 monclient(hunting): my global_id is 4365 2024-03-25T14:30:10.787 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.782+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c0b7520 con 0x7fa17c1430c0 2024-03-25T14:30:10.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.784+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660948745 0 0) 0x7fa160004eb0 con 0x7fa17c13f710 2024-03-25T14:30:10.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.785+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa17c136ad0 con 0x7fa17c13f710 2024-03-25T14:30:10.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.785+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388925823 0 0) 0x7fa17c135810 con 0x7fa17c0a98f0 2024-03-25T14:30:10.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.785+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa160004eb0 con 0x7fa17c0a98f0 2024-03-25T14:30:10.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa16c0019d0 con 0x7fa17c13f710 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): dump: 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:10.792 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa178001a50 con 0x7fa17c0a98f0 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.787+0000 7fa172ffd640 10 monclient(hunting): dump: 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:10.793 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:10.794 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1693416330 0 0) 0x7fa17c136ad0 con 0x7fa17c13f710 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 >> v1:172.21.15.121:6789/0 conn(0x7fa17c0a98f0 legacy=0x7fa17c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 >> v1:172.21.15.26:6789/0 conn(0x7fa17c1430c0 legacy=0x7fa17c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 1 monclient: found mon.c 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:10.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0b6180 con 0x7fa17c13f710 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 10 monclient: _finish_auth 0 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 5 monclient: authenticate success, global_id 4328 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 10 monclient: _renew_subs 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c0b7a40 con 0x7fa17c13f710 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 10 monclient: _renew_subs 2024-03-25T14:30:10.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:30:10.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c131840 con 0x7fa17c13f710 2024-03-25T14:30:10.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.792+0000 7fa1845cf640 1 librados: init done 2024-03-25T14:30:10.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.793+0000 7fa1845cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:10.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.793+0000 7fa1845cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:10.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.793+0000 7fa1845cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:30:10.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:10.793+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c136f60 con 0x7fa17c13f710 2024-03-25T14:30:11.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.015+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa16c0035f0 con 0x7fa17c13f710 2024-03-25T14:30:11.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.015+0000 7fa172ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:11.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.015+0000 7fa180b68640 4 set_mon_vals no callback set 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.103+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa16c0038a0 con 0x7fa17c13f710 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.103+0000 7fa172ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.103+0000 7fa172ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.103+0000 7fa172ffd640 10 monclient: dump: 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:11.108 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:11.109 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:11.109 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:11.109 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:11.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.104+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fa16c003a80 con 0x7fa17c13f710 2024-03-25T14:30:11.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.105+0000 7fa18236b640 10 monclient: get_auth_request con 0x7fa16006df00 auth_method 0 2024-03-25T14:30:11.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.105+0000 7fa172ffd640 10 client.4328.objecter ms_handle_connect 0x7fa16006df00 2024-03-25T14:30:11.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fa16c0534f0 con 0x7fa17c13f710 2024-03-25T14:30:11.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 10 client.4328.objecter ms_dispatch 0x7fa17c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:11.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:11.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:11.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T14:30:11.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa16c04f000 con 0x7fa17c13f710 2024-03-25T14:30:11.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:11.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.111+0000 7fa172ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:11.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.337+0000 7fa1845cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:11.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.337+0000 7fa1845cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:11.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.337+0000 7fa1845cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:30:11.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.337+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa17c0b6310 con 0x7fa17c13f710 2024-03-25T14:30:11.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.394+0000 7fa172ffd640 1 -- 172.21.15.26:0/1695547154 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fa16c050f30 con 0x7fa17c13f710 2024-03-25T14:30:11.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.394+0000 7fa172ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:11.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.394+0000 7fa172ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:11.400 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:11.400 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 10 librados: watch_flush enter 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 10 librados: watch_flush exit 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 10 client.4328.objecter successfully canceled tick 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 >> v1:172.21.15.121:6800/288813066 conn(0x7fa16006df00 legacy=0x7fa1600703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 10 monclient: shutdown 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:11.402 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 >> v1:172.21.15.26:6790/0 conn(0x7fa17c13f710 legacy=0x7fa17c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:11.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 shutdown_connections 2024-03-25T14:30:11.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 >> 172.21.15.26:0/1695547154 conn(0x7fa17c0b7ea0 msgr2=0x7fa17c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:11.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 shutdown_connections 2024-03-25T14:30:11.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 -- 172.21.15.26:0/1695547154 wait complete. 2024-03-25T14:30:11.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:11.397+0000 7fa1845cf640 1 librados: shutdown 2024-03-25T14:30:11.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:15.983 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:34.843 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.000 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:34.861 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:34.905 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.045 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:34.922 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.054 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:34.915 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.056 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:34.932 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.063 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:34.924 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.085 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:34.962 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:16.112 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:34.973 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:18.222 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:18.339 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.335+0000 7ff0dc4fb640 10 monclient: get_monmap_and_config 2024-03-25T14:30:18.340 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.335+0000 7ff0dc4fb640 10 monclient: build_initial_monmap 2024-03-25T14:30:18.340 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.335+0000 7ff0dc4fb640 10 monclient: monmap: 2024-03-25T14:30:18.340 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:18.340 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.340 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:18.336061+0000 2024-03-25T14:30:18.340 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:18.336061+0000 2024-03-25T14:30:18.341 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:18.341 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.341 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:18.341 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:18.341 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:18.341 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 1 Processor -- start 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 1 -- start start 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 10 monclient: init 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff0d40b10d0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:18.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.336+0000 7ff0dc4fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff0d40b4360 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.337+0000 7ff0dc4fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d40a9470 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.337+0000 7ff0dc4fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40a9ab0 con 0x7ff0d40b4360 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.337+0000 7ff0dc4fb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40b2400 con 0x7ff0d40b10d0 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.337+0000 7ff0dc4fb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40aa2a0 con 0x7ff0d40a9470 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.337+0000 7ff0dc4fb640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.337+0000 7ff0dc4fb640 10 monclient(hunting): authenticate will time out at 1703.439683s 2024-03-25T14:30:18.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.353+0000 7ff0d9a96640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff0d40b4360 0x7ff0d40b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57594/0 (socket says 172.21.15.26:57594) 2024-03-25T14:30:18.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.353+0000 7ff0d9a96640 1 -- 172.21.15.26:0/2226500846 learned_addr learned my addr 172.21.15.26:0/2226500846 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:18.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.372+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516359106 0 0) 0x7ff0d40a9ab0 con 0x7ff0d40b4360 2024-03-25T14:30:18.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.372+0000 7ff0d9295640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.372+0000 7ff0d9295640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.372+0000 7ff0d9295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.372+0000 7ff0d9295640 10 monclient(hunting): my global_id is 4368 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.372+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c0003520 con 0x7ff0d40b4360 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.374+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432120742 0 0) 0x7ff0d40aa2a0 con 0x7ff0d40a9470 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.374+0000 7ff0d9295640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.374+0000 7ff0d9295640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.374+0000 7ff0d9295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.374+0000 7ff0d9295640 10 monclient(hunting): my global_id is 4331 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.374+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d40a9ab0 con 0x7ff0d40a9470 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.375+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474503232 0 0) 0x7ff0d40b2400 con 0x7ff0d40b10d0 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.375+0000 7ff0d9295640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.375+0000 7ff0d9295640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.375+0000 7ff0d9295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.375+0000 7ff0d9295640 10 monclient(hunting): my global_id is 4357 2024-03-25T14:30:18.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.375+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d40aa2a0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974058344 0 0) 0x7ff0d40a9ab0 con 0x7ff0d40a9470 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 >> v1:172.21.15.121:6789/0 conn(0x7ff0d40b10d0 legacy=0x7ff0d40b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 >> v1:172.21.15.26:6789/0 conn(0x7ff0d40b4360 legacy=0x7ff0d40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: _finish_hunting 0 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 1 monclient: found mon.noname-b 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40b14b0 con 0x7ff0d40a9470 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: _finish_auth 0 2024-03-25T14:30:18.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: _check_auth_tickets 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff0cc0019f0 con 0x7ff0d40a9470 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0dc4fb640 5 monclient: authenticate success, global_id 4331 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0dc4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0d9295640 10 monclient: dump: 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.387 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:18.388 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:18.388 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:18.388 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.388 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.381+0000 7ff0dc4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0cc001b50 con 0x7ff0d40a9470 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0d9295640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0d9295640 1 -- 172.21.15.26:0/2226500846 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff0cc001f60 con 0x7ff0d40a9470 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0dc4fb640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0d9295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0dc4fb640 4 set_mon_vals no callback set 2024-03-25T14:30:18.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0d9295640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.393+0000 7ff0d9295640 10 monclient: dump: 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 10 monclient: shutdown 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/2226500846 >> v1:172.21.15.26:6790/0 conn(0x7ff0d40a9470 legacy=0x7ff0d414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/2226500846 shutdown_connections 2024-03-25T14:30:18.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/2226500846 >> 172.21.15.26:0/2226500846 conn(0x7ff0d40617a0 msgr2=0x7ff0d4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:18.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/2226500846 shutdown_connections 2024-03-25T14:30:18.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.394+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/2226500846 wait complete. 2024-03-25T14:30:18.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.395+0000 7ff0dc4fb640 10 monclient: build_initial_monmap 2024-03-25T14:30:18.403 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.395+0000 7ff0dc4fb640 10 monclient: monmap: 2024-03-25T14:30:18.403 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:18.403 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:18.403 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:18.403 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.395+0000 7ff0dc4fb640 1 librados: starting msgr at 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.395+0000 7ff0dc4fb640 1 librados: starting objecter 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.395+0000 7ff0dc4fb640 1 Processor -- start 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 1 -- start start 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 1 librados: setting wanted keys 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 1 librados: calling monclient init 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient: init 2024-03-25T14:30:18.404 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff0d40a9470 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff0d40b10d0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d40b4360 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d413f940 con 0x7ff0d40b10d0 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d4131ef0 con 0x7ff0d40b4360 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d41db920 con 0x7ff0d40a9470 2024-03-25T14:30:18.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:18.406 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.396+0000 7ff0dc4fb640 10 monclient(hunting): authenticate will time out at 1703.499378s 2024-03-25T14:30:18.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.407+0000 7ff0daa98640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff0d40b4360 0x7ff0d41db210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60238/0 (socket says 172.21.15.26:60238) 2024-03-25T14:30:18.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.407+0000 7ff0daa98640 1 -- 172.21.15.26:0/3335666684 learned_addr learned my addr 172.21.15.26:0/3335666684 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:18.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.408+0000 7ff0caffd640 10 client.?.objecter ms_handle_connect 0x7ff0d40b4360 2024-03-25T14:30:18.412 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.408+0000 7ff0caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:18.417 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.412+0000 7ff0caffd640 10 client.?.objecter ms_handle_connect 0x7ff0d40b10d0 2024-03-25T14:30:18.417 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.412+0000 7ff0caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:18.418 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.414+0000 7ff0caffd640 10 client.?.objecter ms_handle_connect 0x7ff0d40a9470 2024-03-25T14:30:18.418 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.414+0000 7ff0caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:18.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.420+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205025888 0 0) 0x7ff0d4131ef0 con 0x7ff0d40b4360 2024-03-25T14:30:18.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.420+0000 7ff0caffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.420+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.420+0000 7ff0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.420+0000 7ff0caffd640 10 monclient(hunting): my global_id is 4360 2024-03-25T14:30:18.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.420+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b8004eb0 con 0x7ff0d40b4360 2024-03-25T14:30:18.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.421+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495189421 0 0) 0x7ff0d41db920 con 0x7ff0d40a9470 2024-03-25T14:30:18.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.421+0000 7ff0caffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.421+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.421+0000 7ff0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.421+0000 7ff0caffd640 10 monclient(hunting): my global_id is 4334 2024-03-25T14:30:18.427 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.421+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d4131ef0 con 0x7ff0d40a9470 2024-03-25T14:30:18.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.429+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034146240 0 0) 0x7ff0d413f940 con 0x7ff0d40b10d0 2024-03-25T14:30:18.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.429+0000 7ff0caffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.429+0000 7ff0caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.438 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.429+0000 7ff0caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.429+0000 7ff0caffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T14:30:18.439 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.429+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d41db920 con 0x7ff0d40b10d0 2024-03-25T14:30:18.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3875916532 0 0) 0x7ff0b8004eb0 con 0x7ff0d40b4360 2024-03-25T14:30:18.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0d413f940 con 0x7ff0d40b4360 2024-03-25T14:30:18.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff0cc001fa0 con 0x7ff0d40b4360 2024-03-25T14:30:18.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 10 monclient(hunting): dump: 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:18.455 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:18.456 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.449+0000 7ff0caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:18.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.455+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528222077 0 0) 0x7ff0d41db920 con 0x7ff0d40b10d0 2024-03-25T14:30:18.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.455+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0b8004eb0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.455+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff0c40017b0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.455+0000 7ff0caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.455+0000 7ff0caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:18.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.455+0000 7ff0caffd640 10 monclient(hunting): dump: 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:18.461 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:18.462 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.457+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 690766440 0 0) 0x7ff0d4131ef0 con 0x7ff0d40a9470 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.457+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0d41db920 con 0x7ff0d40a9470 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.457+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff0d0001870 con 0x7ff0d40a9470 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.457+0000 7ff0caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.457+0000 7ff0caffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.457+0000 7ff0caffd640 10 monclient(hunting): dump: 2024-03-25T14:30:18.463 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:18.464 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3844567361 0 0) 0x7ff0b8004eb0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 >> v1:172.21.15.26:6790/0 conn(0x7ff0d40a9470 legacy=0x7ff0d41317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 >> v1:172.21.15.121:6789/0 conn(0x7ff0d40b4360 legacy=0x7ff0d41db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 10 monclient: _finish_hunting 0 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 1 monclient: found mon.a 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d4130d90 con 0x7ff0d40b10d0 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 10 monclient: _finish_auth 0 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 10 monclient: _check_auth_tickets 2024-03-25T14:30:18.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.470+0000 7ff0caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 5 monclient: authenticate success, global_id 4371 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 10 monclient: _renew_subs 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d4131070 con 0x7ff0d40b10d0 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 10 monclient: _renew_subs 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:30:18.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d4141740 con 0x7ff0d40b10d0 2024-03-25T14:30:18.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.471+0000 7ff0dc4fb640 1 librados: init done 2024-03-25T14:30:18.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.472+0000 7ff0dc4fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:18.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.472+0000 7ff0dc4fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:18.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.472+0000 7ff0dc4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:30:18.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.472+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d4131260 con 0x7ff0d40b10d0 2024-03-25T14:30:18.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0c4001cc0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff0c4004980 con 0x7ff0d40b10d0 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 10 monclient: dump: 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:18.789 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:18.790 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:18.790 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:18.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0d8a94640 4 set_mon_vals no callback set 2024-03-25T14:30:18.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.784+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ff0c401a990 con 0x7ff0d40b10d0 2024-03-25T14:30:18.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.785+0000 7ff0da297640 10 monclient: get_auth_request con 0x7ff0b806e160 auth_method 0 2024-03-25T14:30:18.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.785+0000 7ff0caffd640 10 client.4371.objecter ms_handle_connect 0x7ff0b806e160 2024-03-25T14:30:18.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7ff0c40546a0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 10 client.4371.objecter ms_dispatch 0x7ff0d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0c40501b0 con 0x7ff0d40b10d0 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:18.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:18.791+0000 7ff0caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:19.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.051+0000 7ff0dc4fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:19.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.052+0000 7ff0dc4fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:19.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.052+0000 7ff0dc4fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:30:19.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.052+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0d41420a0 con 0x7ff0d40b10d0 2024-03-25T14:30:19.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.073+0000 7ff0caffd640 1 -- 172.21.15.26:0/3335666684 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7ff0d41420a0 con 0x7ff0d40b10d0 2024-03-25T14:30:19.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.073+0000 7ff0caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:19.078 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.073+0000 7ff0caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:19.079 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:19.079 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:19.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 10 librados: watch_flush enter 2024-03-25T14:30:19.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 10 librados: watch_flush exit 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 10 client.4371.objecter successfully canceled tick 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 >> v1:172.21.15.121:6800/288813066 conn(0x7ff0b806e160 legacy=0x7ff0b8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 10 monclient: shutdown 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 >> v1:172.21.15.26:6789/0 conn(0x7ff0d40b10d0 legacy=0x7ff0d41dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 shutdown_connections 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 >> 172.21.15.26:0/3335666684 conn(0x7ff0d40617a0 msgr2=0x7ff0d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.076+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 shutdown_connections 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.077+0000 7ff0dc4fb640 1 -- 172.21.15.26:0/3335666684 wait complete. 2024-03-25T14:30:19.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:19.077+0000 7ff0dc4fb640 1 librados: shutdown 2024-03-25T14:30:19.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:25.896 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:26.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.014+0000 7f0c3abb5640 10 monclient: get_monmap_and_config 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.014+0000 7f0c3abb5640 10 monclient: build_initial_monmap 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.014+0000 7f0c3abb5640 10 monclient: monmap: 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:26.014911+0000 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:26.014911+0000 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:26.019 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.014+0000 7f0c3abb5640 1 Processor -- start 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.014+0000 7f0c3abb5640 1 -- start start 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.014+0000 7f0c3abb5640 10 monclient: init 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.015+0000 7f0c3abb5640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.015+0000 7f0c3abb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.015+0000 7f0c3abb5640 10 monclient(hunting): picked mon.noname-b con 0x7f0c340a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.015+0000 7f0c3abb5640 10 monclient(hunting): picked mon.noname-c con 0x7f0c3413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.015+0000 7f0c3abb5640 10 monclient(hunting): picked mon.noname-a con 0x7f0c341430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:26.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.016+0000 7f0c3abb5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34142520 con 0x7f0c341430c0 2024-03-25T14:30:26.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.016+0000 7f0c3abb5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340b28a0 con 0x7f0c3413f710 2024-03-25T14:30:26.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.016+0000 7f0c3abb5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340b2a30 con 0x7f0c340a98f0 2024-03-25T14:30:26.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.016+0000 7f0c3abb5640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:26.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.016+0000 7f0c3abb5640 10 monclient(hunting): authenticate will time out at 1711.118757s 2024-03-25T14:30:26.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.023+0000 7f0c33fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0c3413f710 0x7f0c341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:36448/0 (socket says 172.21.15.26:36448) 2024-03-25T14:30:26.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.023+0000 7f0c33fff640 1 -- 172.21.15.26:0/128002814 learned_addr learned my addr 172.21.15.26:0/128002814 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.035+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912299722 0 0) 0x7f0c340b28a0 con 0x7f0c3413f710 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.035+0000 7f0c337fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.035+0000 7f0c337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.035+0000 7f0c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.035+0000 7f0c337fe640 10 monclient(hunting): my global_id is 4363 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.035+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c24003520 con 0x7f0c3413f710 2024-03-25T14:30:26.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.038+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642441885 0 0) 0x7f0c34142520 con 0x7f0c341430c0 2024-03-25T14:30:26.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.038+0000 7f0c337fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.038+0000 7f0c337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.038+0000 7f0c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.038+0000 7f0c337fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T14:30:26.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.038+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c340b28a0 con 0x7f0c341430c0 2024-03-25T14:30:26.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.039+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916414895 0 0) 0x7f0c340b2a30 con 0x7f0c340a98f0 2024-03-25T14:30:26.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.039+0000 7f0c337fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.039+0000 7f0c337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.039+0000 7f0c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.039+0000 7f0c337fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T14:30:26.044 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.039+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c34142520 con 0x7f0c340a98f0 2024-03-25T14:30:26.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061989360 0 0) 0x7f0c24003520 con 0x7f0c3413f710 2024-03-25T14:30:26.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 >> v1:172.21.15.26:6790/0 conn(0x7f0c340a98f0 legacy=0x7f0c340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 >> v1:172.21.15.26:6789/0 conn(0x7f0c341430c0 legacy=0x7f0c3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: _finish_hunting 0 2024-03-25T14:30:26.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 1 monclient: found mon.noname-c 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340b44c0 con 0x7f0c3413f710 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: _finish_auth 0 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: _check_auth_tickets 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0c20001690 con 0x7f0c3413f710 2024-03-25T14:30:26.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c3abb5640 5 monclient: authenticate success, global_id 4363 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c3abb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c337fe640 10 monclient: dump: 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:26.047 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.040+0000 7f0c3abb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c200017f0 con 0x7f0c3413f710 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c337fe640 1 -- 172.21.15.26:0/128002814 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0c20001c00 con 0x7f0c3413f710 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c3abb5640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c3abb5640 4 set_mon_vals no callback set 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.044+0000 7f0c337fe640 10 monclient: dump: 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.049 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.050 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.050 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:26.050 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:26.050 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:26.050 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:26.050 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 10 monclient: shutdown 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 1 -- 172.21.15.26:0/128002814 >> v1:172.21.15.121:6789/0 conn(0x7f0c3413f710 legacy=0x7f0c341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 1 -- 172.21.15.26:0/128002814 shutdown_connections 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 1 -- 172.21.15.26:0/128002814 >> 172.21.15.26:0/128002814 conn(0x7f0c340b7ea0 msgr2=0x7f0c340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 1 -- 172.21.15.26:0/128002814 shutdown_connections 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.045+0000 7f0c3abb5640 1 -- 172.21.15.26:0/128002814 wait complete. 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.046+0000 7f0c3abb5640 10 monclient: build_initial_monmap 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.046+0000 7f0c3abb5640 10 monclient: monmap: 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.046+0000 7f0c3abb5640 1 librados: starting msgr at 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.046+0000 7f0c3abb5640 1 librados: starting objecter 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.046+0000 7f0c3abb5640 1 Processor -- start 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.047+0000 7f0c3abb5640 1 -- start start 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.047+0000 7f0c3abb5640 1 librados: setting wanted keys 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.047+0000 7f0c3abb5640 1 librados: calling monclient init 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.047+0000 7f0c3abb5640 10 monclient: init 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient(hunting): picked mon.noname-b con 0x7f0c340a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient(hunting): picked mon.noname-a con 0x7f0c3413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient(hunting): picked mon.noname-c con 0x7f0c341430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34139a40 con 0x7f0c3413f710 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34133f20 con 0x7f0c340a98f0 2024-03-25T14:30:26.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340ab950 con 0x7f0c341430c0 2024-03-25T14:30:26.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:26.056 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.048+0000 7f0c3abb5640 10 monclient(hunting): authenticate will time out at 1711.150792s 2024-03-25T14:30:26.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.066+0000 7f0c33fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0c3413f710 0x7f0c34133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34058/0 (socket says 172.21.15.26:34058) 2024-03-25T14:30:26.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.066+0000 7f0c33fff640 1 -- 172.21.15.26:0/1484924219 learned_addr learned my addr 172.21.15.26:0/1484924219 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:26.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.068+0000 7f0c317fa640 10 client.?.objecter ms_handle_connect 0x7f0c341430c0 2024-03-25T14:30:26.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.068+0000 7f0c317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:26.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.069+0000 7f0c317fa640 10 client.?.objecter ms_handle_connect 0x7f0c340a98f0 2024-03-25T14:30:26.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.069+0000 7f0c317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:26.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.070+0000 7f0c317fa640 10 client.?.objecter ms_handle_connect 0x7f0c3413f710 2024-03-25T14:30:26.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.070+0000 7f0c317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462560933 0 0) 0x7f0c340ab950 con 0x7f0c341430c0 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): my global_id is 4340 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c18004eb0 con 0x7f0c341430c0 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891950807 0 0) 0x7f0c34133f20 con 0x7f0c340a98f0 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.083 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.084 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 10 monclient(hunting): my global_id is 4366 2024-03-25T14:30:26.084 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.078+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c340ab950 con 0x7f0c340a98f0 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472813562 0 0) 0x7f0c340ab950 con 0x7f0c340a98f0 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c34133f20 con 0x7f0c340a98f0 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0c28001870 con 0x7f0c340a98f0 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 10 monclient(hunting): dump: 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.093 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.094 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:26.094 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:26.094 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:26.094 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:26.094 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.088+0000 7f0c317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:26.100 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 498843700 0 0) 0x7f0c18004eb0 con 0x7f0c341430c0 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c340ab950 con 0x7f0c341430c0 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0c2c001bd0 con 0x7f0c341430c0 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient(hunting): dump: 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:26.101 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2929281723 0 0) 0x7f0c34133f20 con 0x7f0c340a98f0 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 >> v1:172.21.15.26:6790/0 conn(0x7f0c341430c0 legacy=0x7f0c34133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 >> v1:172.21.15.26:6789/0 conn(0x7f0c3413f710 legacy=0x7f0c34133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient: _finish_hunting 0 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 monclient: found mon.b 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340abc30 con 0x7f0c340a98f0 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient: _finish_auth 0 2024-03-25T14:30:26.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 10 monclient: _check_auth_tickets 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c3abb5640 5 monclient: authenticate success, global_id 4366 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c3abb5640 10 monclient: _renew_subs 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c3abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.095+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c340abf10 con 0x7f0c340a98f0 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.096+0000 7f0c3abb5640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.096+0000 7f0c3abb5640 10 monclient: _renew_subs 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.096+0000 7f0c3abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.096+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c34139370 con 0x7f0c340a98f0 2024-03-25T14:30:26.103 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.096+0000 7f0c3abb5640 1 librados: init done 2024-03-25T14:30:26.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.097+0000 7f0c3abb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:26.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.097+0000 7f0c3abb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:26.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.097+0000 7f0c3abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:26.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.097+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c340627b0 con 0x7f0c340a98f0 2024-03-25T14:30:26.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c28003610 con 0x7f0c340a98f0 2024-03-25T14:30:26.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:26.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0c28001d80 con 0x7f0c340a98f0 2024-03-25T14:30:26.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c317fa640 10 monclient: dump: 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:26.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.375+0000 7f0c32ffd640 4 set_mon_vals no callback set 2024-03-25T14:30:26.382 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.376+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f0c28003b20 con 0x7f0c340a98f0 2024-03-25T14:30:26.382 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.377+0000 7f0c33fff640 10 monclient: get_auth_request con 0x7f0c1806de90 auth_method 0 2024-03-25T14:30:26.382 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.377+0000 7f0c317fa640 10 client.4366.objecter ms_handle_connect 0x7f0c1806de90 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f0c28053530 con 0x7f0c340a98f0 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 10 client.4366.objecter ms_dispatch 0x7f0c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c2804f040 con 0x7f0c340a98f0 2024-03-25T14:30:26.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:26.388 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.382+0000 7f0c317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:26.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.629+0000 7f0c3abb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:26.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.629+0000 7f0c3abb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:26.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.629+0000 7f0c3abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:26.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.629+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c340b7180 con 0x7f0c340a98f0 2024-03-25T14:30:26.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.651+0000 7f0c317fa640 1 -- 172.21.15.26:0/1484924219 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f0c28050f70 con 0x7f0c340a98f0 2024-03-25T14:30:26.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.651+0000 7f0c317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:26.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.651+0000 7f0c317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:26.657 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:26.658 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 10 librados: watch_flush enter 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 10 librados: watch_flush exit 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 10 client.4366.objecter successfully canceled tick 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 >> v1:172.21.15.121:6800/288813066 conn(0x7f0c1806de90 legacy=0x7f0c18070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 10 monclient: shutdown 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:26.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.653+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 >> v1:172.21.15.121:6789/0 conn(0x7f0c340a98f0 legacy=0x7f0c34138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.654+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 shutdown_connections 2024-03-25T14:30:26.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.654+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 >> 172.21.15.26:0/1484924219 conn(0x7f0c340b7ea0 msgr2=0x7f0c34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:26.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.654+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 shutdown_connections 2024-03-25T14:30:26.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.654+0000 7f0c3abb5640 1 -- 172.21.15.26:0/1484924219 wait complete. 2024-03-25T14:30:26.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:26.654+0000 7f0c3abb5640 1 librados: shutdown 2024-03-25T14:30:26.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:33.478 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:33.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.590+0000 7fa31ab54640 10 monclient: get_monmap_and_config 2024-03-25T14:30:33.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.590+0000 7fa31ab54640 10 monclient: build_initial_monmap 2024-03-25T14:30:33.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.590+0000 7fa31ab54640 10 monclient: monmap: 2024-03-25T14:30:33.596 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:33.592768+0000 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:33.592768+0000 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.591+0000 7fa31ab54640 1 Processor -- start 2024-03-25T14:30:33.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.591+0000 7fa31ab54640 1 -- start start 2024-03-25T14:30:33.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.591+0000 7fa31ab54640 10 monclient: init 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient(hunting): picked mon.noname-a con 0x7fa3140a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient(hunting): picked mon.noname-c con 0x7fa31413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient(hunting): picked mon.noname-b con 0x7fa3141430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa314142520 con 0x7fa3140a98f0 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140b28a0 con 0x7fa31413f710 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140b2a30 con 0x7fa3141430c0 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.592+0000 7fa31ab54640 10 monclient(hunting): authenticate will time out at 1718.696049s 2024-03-25T14:30:33.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.602+0000 7fa313fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa31413f710 0x7fa3141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:34786/0 (socket says 172.21.15.26:34786) 2024-03-25T14:30:33.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.602+0000 7fa313fff640 1 -- 172.21.15.26:0/1133608836 learned_addr learned my addr 172.21.15.26:0/1133608836 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:33.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.616+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250121805 0 0) 0x7fa3140b28a0 con 0x7fa31413f710 2024-03-25T14:30:33.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.616+0000 7fa3137fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.616+0000 7fa3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.616+0000 7fa3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.616+0000 7fa3137fe640 10 monclient(hunting): my global_id is 4369 2024-03-25T14:30:33.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.616+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304003520 con 0x7fa31413f710 2024-03-25T14:30:33.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.627+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749963699 0 0) 0x7fa314142520 con 0x7fa3140a98f0 2024-03-25T14:30:33.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.627+0000 7fa3137fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.627+0000 7fa3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.627+0000 7fa3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.627+0000 7fa3137fe640 10 monclient(hunting): my global_id is 4380 2024-03-25T14:30:33.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.627+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3140b28a0 con 0x7fa3140a98f0 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216490246 0 0) 0x7fa304003520 con 0x7fa31413f710 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 >> v1:172.21.15.26:6790/0 conn(0x7fa3141430c0 legacy=0x7fa31414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 >> v1:172.21.15.26:6789/0 conn(0x7fa3140a98f0 legacy=0x7fa3140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: _finish_hunting 0 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 1 monclient: found mon.noname-c 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140b44c0 con 0x7fa31413f710 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: _finish_auth 0 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: _check_auth_tickets 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa300001690 con 0x7fa31413f710 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa31ab54640 5 monclient: authenticate success, global_id 4369 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa31ab54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa3137fe640 10 monclient: dump: 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:33.638 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.629+0000 7fa31ab54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3000017f0 con 0x7fa31413f710 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa3137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa3137fe640 1 -- 172.21.15.26:0/1133608836 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa300001c00 con 0x7fa31413f710 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa31ab54640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa31ab54640 4 set_mon_vals no callback set 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.637+0000 7fa3137fe640 10 monclient: dump: 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 10 monclient: shutdown 2024-03-25T14:30:33.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:33.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 1 -- 172.21.15.26:0/1133608836 >> v1:172.21.15.121:6789/0 conn(0x7fa31413f710 legacy=0x7fa3141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 1 -- 172.21.15.26:0/1133608836 shutdown_connections 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 1 -- 172.21.15.26:0/1133608836 >> 172.21.15.26:0/1133608836 conn(0x7fa3140b7ea0 msgr2=0x7fa3140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 1 -- 172.21.15.26:0/1133608836 shutdown_connections 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.638+0000 7fa31ab54640 1 -- 172.21.15.26:0/1133608836 wait complete. 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 10 monclient: build_initial_monmap 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 10 monclient: monmap: 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:33.648 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 1 librados: starting msgr at 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 1 librados: starting objecter 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 1 Processor -- start 2024-03-25T14:30:33.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 1 -- start start 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 1 librados: setting wanted keys 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 1 librados: calling monclient init 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.639+0000 7fa31ab54640 10 monclient: init 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient(hunting): picked mon.noname-a con 0x7fa3140a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient(hunting): picked mon.noname-c con 0x7fa31413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient(hunting): picked mon.noname-b con 0x7fa3141430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa314138a50 con 0x7fa3140a98f0 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140ab9e0 con 0x7fa3141430c0 2024-03-25T14:30:33.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3141d7880 con 0x7fa31413f710 2024-03-25T14:30:33.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:33.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.640+0000 7fa31ab54640 10 monclient(hunting): authenticate will time out at 1718.744339s 2024-03-25T14:30:33.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.650+0000 7fa3188f0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa3140a98f0 0x7fa314138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:49126/0 (socket says 172.21.15.26:49126) 2024-03-25T14:30:33.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.650+0000 7fa3188f0640 1 -- 172.21.15.26:0/28585725 learned_addr learned my addr 172.21.15.26:0/28585725 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:33.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.651+0000 7fa3117fa640 10 client.?.objecter ms_handle_connect 0x7fa3140a98f0 2024-03-25T14:30:33.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.651+0000 7fa3117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:33.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.654+0000 7fa3117fa640 10 client.?.objecter ms_handle_connect 0x7fa3141430c0 2024-03-25T14:30:33.660 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.654+0000 7fa3117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:33.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.656+0000 7fa3117fa640 10 client.?.objecter ms_handle_connect 0x7fa31413f710 2024-03-25T14:30:33.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.656+0000 7fa3117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:33.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.671+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882074962 0 0) 0x7fa3140ab9e0 con 0x7fa3141430c0 2024-03-25T14:30:33.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.671+0000 7fa3117fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.671+0000 7fa3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.671+0000 7fa3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.671+0000 7fa3117fa640 10 monclient(hunting): my global_id is 4372 2024-03-25T14:30:33.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.671+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f8004eb0 con 0x7fa3141430c0 2024-03-25T14:30:33.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.676+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553495449 0 0) 0x7fa3141d7880 con 0x7fa31413f710 2024-03-25T14:30:33.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.677+0000 7fa3117fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.677+0000 7fa3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.677+0000 7fa3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.677+0000 7fa3117fa640 10 monclient(hunting): my global_id is 4346 2024-03-25T14:30:33.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.677+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3140ab9e0 con 0x7fa31413f710 2024-03-25T14:30:33.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.679+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022575065 0 0) 0x7fa314138a50 con 0x7fa3140a98f0 2024-03-25T14:30:33.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.679+0000 7fa3117fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.679+0000 7fa3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.679+0000 7fa3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.679+0000 7fa3117fa640 10 monclient(hunting): my global_id is 4383 2024-03-25T14:30:33.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.679+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3141d7880 con 0x7fa3140a98f0 2024-03-25T14:30:33.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864234754 0 0) 0x7fa2f8004eb0 con 0x7fa3141430c0 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa314138a50 con 0x7fa3141430c0 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa30c001b90 con 0x7fa3141430c0 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 10 monclient(hunting): dump: 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:33.691 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:33.692 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:33.692 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:33.692 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.685+0000 7fa3117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.711+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804582291 0 0) 0x7fa3140ab9e0 con 0x7fa31413f710 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.711+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f8004eb0 con 0x7fa31413f710 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.711+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa300001b00 con 0x7fa31413f710 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.711+0000 7fa3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.711+0000 7fa3117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.711+0000 7fa3117fa640 10 monclient(hunting): dump: 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:33.717 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:33.718 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:33.718 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:33.718 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.714+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3195638218 0 0) 0x7fa3141d7880 con 0x7fa3140a98f0 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.714+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3140ab9e0 con 0x7fa3140a98f0 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.714+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa3080019b0 con 0x7fa3140a98f0 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.714+0000 7fa3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.714+0000 7fa3117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.714+0000 7fa3117fa640 10 monclient(hunting): dump: 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.720 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:33.721 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:33.721 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:33.721 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:33.721 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:33.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 796425817 0 0) 0x7fa314138a50 con 0x7fa3141430c0 2024-03-25T14:30:33.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 >> v1:172.21.15.26:6790/0 conn(0x7fa31413f710 legacy=0x7fa3141d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 >> v1:172.21.15.26:6789/0 conn(0x7fa3140a98f0 legacy=0x7fa314138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 10 monclient: _finish_hunting 0 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 1 monclient: found mon.b 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3141d84b0 con 0x7fa3141430c0 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.723+0000 7fa3117fa640 10 monclient: _finish_auth 0 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa3117fa640 10 monclient: _check_auth_tickets 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa3117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 5 monclient: authenticate success, global_id 4372 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 10 monclient: _renew_subs 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3141d51f0 con 0x7fa3141430c0 2024-03-25T14:30:33.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 10 monclient: _renew_subs 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa314138e60 con 0x7fa3141430c0 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.724+0000 7fa31ab54640 1 librados: init done 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.725+0000 7fa31ab54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.725+0000 7fa31ab54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.725+0000 7fa31ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:33.731 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:33.725+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3141d76e0 con 0x7fa3141430c0 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa30c002080 con 0x7fa3141430c0 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa3117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa30c003970 con 0x7fa3141430c0 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa3117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa3117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa3117fa640 10 monclient: dump: 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:34.016 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:34.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.010+0000 7fa312ffd640 4 set_mon_vals no callback set 2024-03-25T14:30:34.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.011+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fa30c003b90 con 0x7fa3141430c0 2024-03-25T14:30:34.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.012+0000 7fa3188f0640 10 monclient: get_auth_request con 0x7fa2f806e0b0 auth_method 0 2024-03-25T14:30:34.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.012+0000 7fa3117fa640 10 client.4372.objecter ms_handle_connect 0x7fa2f806e0b0 2024-03-25T14:30:34.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.017+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fa30c053700 con 0x7fa3141430c0 2024-03-25T14:30:34.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.017+0000 7fa3117fa640 10 client.4372.objecter ms_dispatch 0x7fa314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:34.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.017+0000 7fa3117fa640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:34.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.017+0000 7fa3117fa640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:34.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.017+0000 7fa3117fa640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T14:30:34.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.018+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa30c04f040 con 0x7fa3141430c0 2024-03-25T14:30:34.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.018+0000 7fa3117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:34.024 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.018+0000 7fa3117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:34.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.264+0000 7fa31ab54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:34.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.264+0000 7fa31ab54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:34.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.264+0000 7fa31ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:34.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.264+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3140627b0 con 0x7fa3141430c0 2024-03-25T14:30:34.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.325+0000 7fa3117fa640 1 -- 172.21.15.26:0/28585725 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fa30c051140 con 0x7fa3141430c0 2024-03-25T14:30:34.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.325+0000 7fa3117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:34.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.325+0000 7fa3117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:34.332 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:34.332 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:34.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 10 librados: watch_flush enter 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 10 librados: watch_flush exit 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 10 client.4372.objecter successfully canceled tick 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 >> v1:172.21.15.121:6800/288813066 conn(0x7fa2f806e0b0 legacy=0x7fa2f8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 10 monclient: shutdown 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.328+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 >> v1:172.21.15.121:6789/0 conn(0x7fa3141430c0 legacy=0x7fa3141e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.329+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 shutdown_connections 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.329+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 >> 172.21.15.26:0/28585725 conn(0x7fa3140b7ea0 msgr2=0x7fa314074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.329+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 shutdown_connections 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.329+0000 7fa31ab54640 1 -- 172.21.15.26:0/28585725 wait complete. 2024-03-25T14:30:34.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:34.329+0000 7fa31ab54640 1 librados: shutdown 2024-03-25T14:30:34.344 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:36.052 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:54.913 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.060 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:54.921 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.093 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:03:54.954 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.110 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:54.987 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.113 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:54.990 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.121 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:54.980 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.128 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:54.989 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.140 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:03:55.017 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:36.163 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:03:55.024 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:41.151 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.259+0000 7f9e2da7b640 10 monclient: get_monmap_and_config 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.259+0000 7f9e2da7b640 10 monclient: build_initial_monmap 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.259+0000 7f9e2da7b640 10 monclient: monmap: 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:41.260241+0000 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:41.260241+0000 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:41.264 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:41.265 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:41.265 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:41.265 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:41.265 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.260+0000 7f9e2da7b640 1 Processor -- start 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.260+0000 7f9e2da7b640 1 -- start start 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.260+0000 7f9e2da7b640 10 monclient: init 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 10 monclient(hunting): picked mon.noname-c con 0x7f9e280a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:41.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:41.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 10 monclient(hunting): picked mon.noname-a con 0x7f9e281430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:41.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e28142520 con 0x7f9e281430c0 2024-03-25T14:30:41.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280b28a0 con 0x7f9e280a98f0 2024-03-25T14:30:41.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.261+0000 7f9e2da7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280b2a30 con 0x7f9e2813f710 2024-03-25T14:30:41.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.262+0000 7f9e2da7b640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:41.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.262+0000 7f9e2da7b640 10 monclient(hunting): authenticate will time out at 1726.364610s 2024-03-25T14:30:41.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.267+0000 7f9e277fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9e280a98f0 0x7f9e280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:33302/0 (socket says 172.21.15.26:33302) 2024-03-25T14:30:41.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.267+0000 7f9e277fe640 1 -- 172.21.15.26:0/338278158 learned_addr learned my addr 172.21.15.26:0/338278158 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:41.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.278+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414676298 0 0) 0x7f9e280b28a0 con 0x7f9e280a98f0 2024-03-25T14:30:41.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.278+0000 7f9e267fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.278+0000 7f9e267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.278+0000 7f9e267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.278+0000 7f9e267fc640 10 monclient(hunting): my global_id is 4375 2024-03-25T14:30:41.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.278+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e10003520 con 0x7f9e280a98f0 2024-03-25T14:30:41.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860724435 0 0) 0x7f9e10003520 con 0x7f9e280a98f0 2024-03-25T14:30:41.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 >> v1:172.21.15.26:6790/0 conn(0x7f9e2813f710 legacy=0x7f9e281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 >> v1:172.21.15.26:6789/0 conn(0x7f9e281430c0 legacy=0x7f9e2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: _finish_hunting 0 2024-03-25T14:30:41.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 1 monclient: found mon.noname-c 2024-03-25T14:30:41.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:41.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:30:41.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280b44c0 con 0x7f9e280a98f0 2024-03-25T14:30:41.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: _finish_auth 0 2024-03-25T14:30:41.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: _check_auth_tickets 2024-03-25T14:30:41.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:41.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9e18001690 con 0x7f9e280a98f0 2024-03-25T14:30:41.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e2da7b640 5 monclient: authenticate success, global_id 4375 2024-03-25T14:30:41.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e2da7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:41.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e267fc640 10 monclient: dump: 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:41.298 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:41.299 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:41.299 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:41.299 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:41.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.289+0000 7f9e2da7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e180017f0 con 0x7f9e280a98f0 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e267fc640 1 -- 172.21.15.26:0/338278158 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9e18001c00 con 0x7f9e280a98f0 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e267fc640 10 monclient: dump: 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:41.324 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e2da7b640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:41.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.319+0000 7f9e2da7b640 4 set_mon_vals no callback set 2024-03-25T14:30:41.328 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 10 monclient: shutdown 2024-03-25T14:30:41.328 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 1 -- 172.21.15.26:0/338278158 >> v1:172.21.15.121:6789/0 conn(0x7f9e280a98f0 legacy=0x7f9e280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 1 -- 172.21.15.26:0/338278158 shutdown_connections 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 1 -- 172.21.15.26:0/338278158 >> 172.21.15.26:0/338278158 conn(0x7f9e280b7ea0 msgr2=0x7f9e280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 1 -- 172.21.15.26:0/338278158 shutdown_connections 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.320+0000 7f9e2da7b640 1 -- 172.21.15.26:0/338278158 wait complete. 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.321+0000 7f9e2da7b640 10 monclient: build_initial_monmap 2024-03-25T14:30:41.329 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.321+0000 7f9e2da7b640 10 monclient: monmap: 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:41.330 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.321+0000 7f9e2da7b640 1 librados: starting msgr at 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.322+0000 7f9e2da7b640 1 librados: starting objecter 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.322+0000 7f9e2da7b640 1 Processor -- start 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.322+0000 7f9e2da7b640 1 -- start start 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.322+0000 7f9e2da7b640 1 librados: setting wanted keys 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.322+0000 7f9e2da7b640 1 librados: calling monclient init 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.322+0000 7f9e2da7b640 10 monclient: init 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:30:41.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient(hunting): picked mon.noname-c con 0x7f9e280a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2813f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient(hunting): picked mon.noname-a con 0x7f9e281430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e28138b20 con 0x7f9e281430c0 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e28137a80 con 0x7f9e2813f710 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e28139880 con 0x7f9e280a98f0 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:41.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.323+0000 7f9e2da7b640 10 monclient(hunting): authenticate will time out at 1726.426017s 2024-03-25T14:30:41.333 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.328+0000 7f9e26ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9e2813f710 0x7f9e28139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:33360/0 (socket says 172.21.15.26:33360) 2024-03-25T14:30:41.333 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.328+0000 7f9e26ffd640 1 -- 172.21.15.26:0/2219425265 learned_addr learned my addr 172.21.15.26:0/2219425265 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:41.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.329+0000 7f9e0ffff640 10 client.?.objecter ms_handle_connect 0x7f9e2813f710 2024-03-25T14:30:41.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.329+0000 7f9e0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:41.336 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.331+0000 7f9e0ffff640 10 client.?.objecter ms_handle_connect 0x7f9e281430c0 2024-03-25T14:30:41.336 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.332+0000 7f9e0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:41.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.336+0000 7f9e0ffff640 10 client.?.objecter ms_handle_connect 0x7f9e280a98f0 2024-03-25T14:30:41.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.336+0000 7f9e0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:41.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.341+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937573712 0 0) 0x7f9e28137a80 con 0x7f9e2813f710 2024-03-25T14:30:41.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.341+0000 7f9e0ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.341+0000 7f9e0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.341+0000 7f9e0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.341+0000 7f9e0ffff640 10 monclient(hunting): my global_id is 4378 2024-03-25T14:30:41.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.341+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e14004eb0 con 0x7f9e2813f710 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3568010263 0 0) 0x7f9e14004eb0 con 0x7f9e2813f710 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e28137a80 con 0x7f9e2813f710 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9e1c001a30 con 0x7f9e2813f710 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 10 monclient(hunting): dump: 2024-03-25T14:30:41.353 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:41.354 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.348+0000 7f9e0ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:41.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826321932 0 0) 0x7f9e28139880 con 0x7f9e280a98f0 2024-03-25T14:30:41.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): my global_id is 4352 2024-03-25T14:30:41.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e14004eb0 con 0x7f9e280a98f0 2024-03-25T14:30:41.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521184152 0 0) 0x7f9e28138b20 con 0x7f9e281430c0 2024-03-25T14:30:41.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 10 monclient(hunting): my global_id is 4389 2024-03-25T14:30:41.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.350+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e28139880 con 0x7f9e281430c0 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2905889636 0 0) 0x7f9e28137a80 con 0x7f9e2813f710 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 >> v1:172.21.15.26:6790/0 conn(0x7f9e280a98f0 legacy=0x7f9e28137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 >> v1:172.21.15.26:6789/0 conn(0x7f9e281430c0 legacy=0x7f9e281e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 1 monclient: found mon.b 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.363+0000 7f9e0ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28139a10 con 0x7f9e2813f710 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e0ffff640 10 monclient: _finish_auth 0 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e0ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:30:41.369 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 5 monclient: authenticate success, global_id 4378 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 10 monclient: _renew_subs 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e281dab00 con 0x7f9e2813f710 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 10 monclient: _renew_subs 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2813b4f0 con 0x7f9e2813f710 2024-03-25T14:30:41.370 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.364+0000 7f9e2da7b640 1 librados: init done 2024-03-25T14:30:41.371 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.365+0000 7f9e2da7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:41.371 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.365+0000 7f9e2da7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:41.371 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.365+0000 7f9e2da7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:41.371 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.365+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e280627b0 con 0x7f9e2813f710 2024-03-25T14:30:41.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e1c001cb0 con 0x7f9e2813f710 2024-03-25T14:30:41.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:41.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9e1c003a60 con 0x7f9e2813f710 2024-03-25T14:30:41.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e0ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e0ffff640 10 monclient: dump: 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:41.751 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:41.752 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:41.752 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:41.752 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.646+0000 7f9e25ffb640 4 set_mon_vals no callback set 2024-03-25T14:30:41.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.647+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f9e1c003c80 con 0x7f9e2813f710 2024-03-25T14:30:41.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.648+0000 7f9e277fe640 10 monclient: get_auth_request con 0x7f9e1406de10 auth_method 0 2024-03-25T14:30:41.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.648+0000 7f9e0ffff640 10 client.4378.objecter ms_handle_connect 0x7f9e1406de10 2024-03-25T14:30:41.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f9e1c0548d0 con 0x7f9e2813f710 2024-03-25T14:30:41.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 10 client.4378.objecter ms_dispatch 0x7f9e28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:41.753 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:41.754 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:41.754 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T14:30:41.754 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e1c0503e0 con 0x7f9e2813f710 2024-03-25T14:30:41.754 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:41.754 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.652+0000 7f9e0ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:41.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.896+0000 7f9e2da7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:41.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.896+0000 7f9e2da7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:41.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.896+0000 7f9e2da7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:41.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.896+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e280b3e80 con 0x7f9e2813f710 2024-03-25T14:30:41.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.973+0000 7f9e0ffff640 1 -- 172.21.15.26:0/2219425265 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f9e1c052310 con 0x7f9e2813f710 2024-03-25T14:30:41.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.973+0000 7f9e0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:41.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.973+0000 7f9e0ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:41.979 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:41.979 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:41.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 10 librados: watch_flush enter 2024-03-25T14:30:41.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 10 librados: watch_flush exit 2024-03-25T14:30:41.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T14:30:41.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 10 client.4378.objecter successfully canceled tick 2024-03-25T14:30:41.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 >> v1:172.21.15.121:6800/288813066 conn(0x7f9e1406de10 legacy=0x7f9e140702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 10 monclient: shutdown 2024-03-25T14:30:41.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:41.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 >> v1:172.21.15.121:6789/0 conn(0x7f9e2813f710 legacy=0x7f9e28139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 shutdown_connections 2024-03-25T14:30:41.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 >> 172.21.15.26:0/2219425265 conn(0x7f9e280b7ea0 msgr2=0x7f9e28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:41.984 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.976+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 shutdown_connections 2024-03-25T14:30:41.984 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.977+0000 7f9e2da7b640 1 -- 172.21.15.26:0/2219425265 wait complete. 2024-03-25T14:30:41.984 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:41.977+0000 7f9e2da7b640 1 librados: shutdown 2024-03-25T14:30:41.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:48.798 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:48.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.905+0000 7f092a42e640 10 monclient: get_monmap_and_config 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.906+0000 7f092a42e640 10 monclient: build_initial_monmap 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.906+0000 7f092a42e640 10 monclient: monmap: 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:48.906871+0000 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:48.906871+0000 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:48.911 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:48.912 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:48.912 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:48.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.906+0000 7f092a42e640 1 Processor -- start 2024-03-25T14:30:48.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 1 -- start start 2024-03-25T14:30:48.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient: init 2024-03-25T14:30:48.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:48.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:30:48.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient(hunting): picked mon.noname-a con 0x7f09240705b0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:48.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient(hunting): picked mon.noname-b con 0x7f09241402d0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:48.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient(hunting): picked mon.noname-c con 0x7f0924149ae0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:48.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09240718c0 con 0x7f09240705b0 2024-03-25T14:30:48.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09240714c0 con 0x7f0924149ae0 2024-03-25T14:30:48.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924072b50 con 0x7f09241402d0 2024-03-25T14:30:48.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:48.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.907+0000 7f092a42e640 10 monclient(hunting): authenticate will time out at 1734.010459s 2024-03-25T14:30:48.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.910+0000 7f0923fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f09240705b0 0x7f092413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57752/0 (socket says 172.21.15.26:57752) 2024-03-25T14:30:48.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.911+0000 7f0923fff640 1 -- 172.21.15.26:0/2487067627 learned_addr learned my addr 172.21.15.26:0/2487067627 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:48.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.924+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034809281 0 0) 0x7f09240718c0 con 0x7f09240705b0 2024-03-25T14:30:48.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.924+0000 7f0922ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.924+0000 7f0922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.924+0000 7f0922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.924+0000 7f0922ffd640 10 monclient(hunting): my global_id is 4392 2024-03-25T14:30:48.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.924+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c003520 con 0x7f09240705b0 2024-03-25T14:30:48.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.930+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402880843 0 0) 0x7f09240714c0 con 0x7f0924149ae0 2024-03-25T14:30:48.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.930+0000 7f0922ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.930+0000 7f0922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.930+0000 7f0922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.930+0000 7f0922ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T14:30:48.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.930+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09240718c0 con 0x7f0924149ae0 2024-03-25T14:30:48.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1907220693 0 0) 0x7f090c003520 con 0x7f09240705b0 2024-03-25T14:30:48.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 >> v1:172.21.15.26:6790/0 conn(0x7f09241402d0 legacy=0x7f0924140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 >> v1:172.21.15.121:6789/0 conn(0x7f0924149ae0 legacy=0x7f092414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:30:48.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 1 monclient: found mon.noname-a 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092413dd10 con 0x7f09240705b0 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: _finish_auth 0 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0914001690 con 0x7f09240705b0 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:30:48.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f0922ffd640 10 monclient: dump: 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f092a42e640 5 monclient: authenticate success, global_id 4392 2024-03-25T14:30:48.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.937+0000 7f092a42e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09140017f0 con 0x7f09240705b0 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f0922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f0922ffd640 1 -- 172.21.15.26:0/2487067627 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0914001c00 con 0x7f09240705b0 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f0922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f0922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f0922ffd640 10 monclient: dump: 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:48.955 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:48.956 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:48.956 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:48.956 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:48.956 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:48.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f092a42e640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:48.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.950+0000 7f092a42e640 4 set_mon_vals no callback set 2024-03-25T14:30:48.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 10 monclient: shutdown 2024-03-25T14:30:48.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:48.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 1 -- 172.21.15.26:0/2487067627 >> v1:172.21.15.26:6789/0 conn(0x7f09240705b0 legacy=0x7f092413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 1 -- 172.21.15.26:0/2487067627 shutdown_connections 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 1 -- 172.21.15.26:0/2487067627 >> 172.21.15.26:0/2487067627 conn(0x7f09240b5d10 msgr2=0x7f09240b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 1 -- 172.21.15.26:0/2487067627 shutdown_connections 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 1 -- 172.21.15.26:0/2487067627 wait complete. 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 10 monclient: build_initial_monmap 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.951+0000 7f092a42e640 10 monclient: monmap: 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:48.959 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 1 librados: starting msgr at 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 1 librados: starting objecter 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 1 Processor -- start 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 1 -- start start 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 1 librados: setting wanted keys 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 1 librados: calling monclient init 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.952+0000 7f092a42e640 10 monclient: init 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:30:48.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient(hunting): picked mon.noname-c con 0x7f09240705b0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient(hunting): picked mon.noname-b con 0x7f09241402d0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient(hunting): picked mon.noname-a con 0x7f0924149ae0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092406c510 con 0x7f0924149ae0 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924137bc0 con 0x7f09241402d0 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924137d50 con 0x7f09240705b0 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:48.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.953+0000 7f092a42e640 10 monclient(hunting): authenticate will time out at 1734.056174s 2024-03-25T14:30:48.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.969+0000 7f09289cb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0924149ae0 0x7f09241d7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57780/0 (socket says 172.21.15.26:57780) 2024-03-25T14:30:48.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.969+0000 7f09289cb640 1 -- 172.21.15.26:0/2906309363 learned_addr learned my addr 172.21.15.26:0/2906309363 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:48.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.969+0000 7f0920ff9640 10 client.?.objecter ms_handle_connect 0x7f0924149ae0 2024-03-25T14:30:48.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.969+0000 7f0920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:48.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.971+0000 7f0920ff9640 10 client.?.objecter ms_handle_connect 0x7f09240705b0 2024-03-25T14:30:48.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.971+0000 7f0920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:48.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.972+0000 7f0920ff9640 10 client.?.objecter ms_handle_connect 0x7f09241402d0 2024-03-25T14:30:48.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.972+0000 7f0920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:48.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.978+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221807139 0 0) 0x7f0924137bc0 con 0x7f09241402d0 2024-03-25T14:30:48.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.978+0000 7f0920ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.978+0000 7f0920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.978+0000 7f0920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.978+0000 7f0920ff9640 10 monclient(hunting): my global_id is 4384 2024-03-25T14:30:48.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.978+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0910004eb0 con 0x7f09241402d0 2024-03-25T14:30:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.988+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701014663 0 0) 0x7f092406c510 con 0x7f0924149ae0 2024-03-25T14:30:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.988+0000 7f0920ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.988+0000 7f0920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.988+0000 7f0920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.988+0000 7f0920ff9640 10 monclient(hunting): my global_id is 4395 2024-03-25T14:30:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.988+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0924137bc0 con 0x7f0924149ae0 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227810023 0 0) 0x7f0910004eb0 con 0x7f09241402d0 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f092406c510 con 0x7f09241402d0 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0918001930 con 0x7f09241402d0 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 10 monclient(hunting): dump: 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:48.995 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:48.996 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:48.996 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:48.996 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.990+0000 7f0920ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:48.998 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.994+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768253669 0 0) 0x7f0924137d50 con 0x7f09240705b0 2024-03-25T14:30:48.998 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.994+0000 7f0920ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.998 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.994+0000 7f0920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.998 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.994+0000 7f0920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.999 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.994+0000 7f0920ff9640 10 monclient(hunting): my global_id is 4358 2024-03-25T14:30:48.999 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:48.994+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0910004eb0 con 0x7f09240705b0 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.014+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007232508 0 0) 0x7f0924137bc0 con 0x7f0924149ae0 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.014+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0924137d50 con 0x7f0924149ae0 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.014+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f091c001b90 con 0x7f0924149ae0 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.014+0000 7f0920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.014+0000 7f0920ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.014+0000 7f0920ff9640 10 monclient(hunting): dump: 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:49.019 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:49.020 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:49.020 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:49.020 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:49.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1196234859 0 0) 0x7f092406c510 con 0x7f09241402d0 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 >> v1:172.21.15.26:6790/0 conn(0x7f09240705b0 legacy=0x7f092406bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 >> v1:172.21.15.26:6789/0 conn(0x7f0924149ae0 legacy=0x7f09241d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 1 monclient: found mon.b 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924137ee0 con 0x7f09241402d0 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 10 monclient: _finish_auth 0 2024-03-25T14:30:49.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f0920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f092a42e640 5 monclient: authenticate success, global_id 4384 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f092a42e640 10 monclient: _renew_subs 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f092a42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.023+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0924148f40 con 0x7f09241402d0 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.024+0000 7f092a42e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.024+0000 7f092a42e640 10 monclient: _renew_subs 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.024+0000 7f092a42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.024+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09241e2fd0 con 0x7f09241402d0 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.024+0000 7f092a42e640 1 librados: init done 2024-03-25T14:30:49.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.025+0000 7f092a42e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:49.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.025+0000 7f092a42e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:49.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.025+0000 7f092a42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:49.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.025+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09240627b0 con 0x7f09241402d0 2024-03-25T14:30:49.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.306+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09180035f0 con 0x7f09241402d0 2024-03-25T14:30:49.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f0920ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0918003900 con 0x7f09241402d0 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f0920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f0920ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f0920ff9640 10 monclient: dump: 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:49.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f09227fc640 4 set_mon_vals no callback set 2024-03-25T14:30:49.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.307+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f09180198d0 con 0x7f09241402d0 2024-03-25T14:30:49.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.308+0000 7f0923fff640 10 monclient: get_auth_request con 0x7f091006df70 auth_method 0 2024-03-25T14:30:49.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.308+0000 7f0920ff9640 10 client.4384.objecter ms_handle_connect 0x7f091006df70 2024-03-25T14:30:49.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.313+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f0918053680 con 0x7f09241402d0 2024-03-25T14:30:49.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.313+0000 7f0920ff9640 10 client.4384.objecter ms_dispatch 0x7f0924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:49.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.313+0000 7f0920ff9640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:49.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.313+0000 7f0920ff9640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:49.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.313+0000 7f0920ff9640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T14:30:49.319 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.314+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f091804efc0 con 0x7f09241402d0 2024-03-25T14:30:49.319 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.314+0000 7f0920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:49.319 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.314+0000 7f0920ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:49.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.568+0000 7f092a42e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:49.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.568+0000 7f092a42e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:49.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.568+0000 7f092a42e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:49.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.568+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f092406c6a0 con 0x7f09241402d0 2024-03-25T14:30:49.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.601+0000 7f0920ff9640 1 -- 172.21.15.26:0/2906309363 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f09180510c0 con 0x7f09241402d0 2024-03-25T14:30:49.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.601+0000 7f0920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:49.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.601+0000 7f0920ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:49.606 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:49.606 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 10 librados: watch_flush enter 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 10 librados: watch_flush exit 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 10 client.4384.objecter successfully canceled tick 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 >> v1:172.21.15.121:6800/288813066 conn(0x7f091006df70 legacy=0x7f0910070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 10 monclient: shutdown 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 >> v1:172.21.15.121:6789/0 conn(0x7f09241402d0 legacy=0x7f09241d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 shutdown_connections 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 >> 172.21.15.26:0/2906309363 conn(0x7f09240b5d10 msgr2=0x7f0924140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:49.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 shutdown_connections 2024-03-25T14:30:49.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 -- 172.21.15.26:0/2906309363 wait complete. 2024-03-25T14:30:49.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:49.604+0000 7f092a42e640 1 librados: shutdown 2024-03-25T14:30:49.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:56.077 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:04:14.938 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.127 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:04:14.987 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.167 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:04:15.027 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.178 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:04:15.055 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.185 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:04:15.061 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.211 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:04:15.087 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.301 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:04:15.161 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.314 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:04:15.175 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.349 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:04:15.210 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.428 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.533+0000 7f500824e640 10 monclient: get_monmap_and_config 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.533+0000 7f500824e640 10 monclient: build_initial_monmap 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.533+0000 7f500824e640 10 monclient: monmap: 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:30:56.534393+0000 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:30:56.534393+0000 2024-03-25T14:30:56.538 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:56.539 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:56.539 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:56.539 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:30:56.539 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:30:56.539 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.534+0000 7f500824e640 1 Processor -- start 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.534+0000 7f500824e640 1 -- start start 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.534+0000 7f500824e640 10 monclient: init 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient(hunting): picked mon.noname-a con 0x7f50000a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient(hunting): picked mon.noname-b con 0x7f500013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:56.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient(hunting): picked mon.noname-c con 0x7f50001430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:56.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5000142520 con 0x7f50000a98f0 2024-03-25T14:30:56.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000b28a0 con 0x7f50001430c0 2024-03-25T14:30:56.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000b2a30 con 0x7f500013f710 2024-03-25T14:30:56.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:56.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.535+0000 7f500824e640 10 monclient(hunting): authenticate will time out at 1741.637829s 2024-03-25T14:30:56.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.548+0000 7f50057e9640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f500013f710 0x7f50001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48970/0 (socket says 172.21.15.26:48970) 2024-03-25T14:30:56.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.548+0000 7f50057e9640 1 -- 172.21.15.26:0/1008546890 learned_addr learned my addr 172.21.15.26:0/1008546890 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:56.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.559+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967193820 0 0) 0x7f50000b2a30 con 0x7f500013f710 2024-03-25T14:30:56.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.559+0000 7f5004fe8640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.559+0000 7f5004fe8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.559+0000 7f5004fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.559+0000 7f5004fe8640 10 monclient(hunting): my global_id is 4361 2024-03-25T14:30:56.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.559+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fec003520 con 0x7f500013f710 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.574+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910096429 0 0) 0x7f5000142520 con 0x7f50000a98f0 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient(hunting): my global_id is 4398 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50000b2a30 con 0x7f50000a98f0 2024-03-25T14:30:56.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3584867552 0 0) 0x7f4fec003520 con 0x7f500013f710 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 >> v1:172.21.15.121:6789/0 conn(0x7f50001430c0 legacy=0x7f500014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 >> v1:172.21.15.26:6789/0 conn(0x7f50000a98f0 legacy=0x7f50000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: _finish_hunting 0 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 monclient: found mon.noname-b 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50000b44c0 con 0x7f500013f710 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: _finish_auth 0 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: _check_auth_tickets 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ff0001690 con 0x7f500013f710 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:56.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f5004fe8640 10 monclient: dump: 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f500824e640 5 monclient: authenticate success, global_id 4361 2024-03-25T14:30:56.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.575+0000 7f500824e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:56.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ff00017f0 con 0x7f500013f710 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f5004fe8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f5004fe8640 1 -- 172.21.15.26:0/1008546890 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ff0001c00 con 0x7f500013f710 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f5004fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f5004fe8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f5004fe8640 10 monclient: dump: 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:56.590 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f500824e640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:56.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.584+0000 7f500824e640 4 set_mon_vals no callback set 2024-03-25T14:30:56.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 10 monclient: shutdown 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 1 -- 172.21.15.26:0/1008546890 >> v1:172.21.15.26:6790/0 conn(0x7f500013f710 legacy=0x7f50001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 1 -- 172.21.15.26:0/1008546890 shutdown_connections 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 1 -- 172.21.15.26:0/1008546890 >> 172.21.15.26:0/1008546890 conn(0x7f50000b7ea0 msgr2=0x7f50000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 1 -- 172.21.15.26:0/1008546890 shutdown_connections 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.585+0000 7f500824e640 1 -- 172.21.15.26:0/1008546890 wait complete. 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 10 monclient: build_initial_monmap 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 10 monclient: monmap: 2024-03-25T14:30:56.594 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 1 librados: starting msgr at 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 1 librados: starting objecter 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 1 Processor -- start 2024-03-25T14:30:56.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 1 -- start start 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 1 librados: setting wanted keys 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 1 librados: calling monclient init 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.586+0000 7f500824e640 10 monclient: init 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient(hunting): picked mon.noname-b con 0x7f50000a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient(hunting): picked mon.noname-a con 0x7f500013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient(hunting): picked mon.noname-c con 0x7f50001430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000ab8b0 con 0x7f500013f710 2024-03-25T14:30:56.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50000abb90 con 0x7f50000a98f0 2024-03-25T14:30:56.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5000137870 con 0x7f50001430c0 2024-03-25T14:30:56.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:56.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.587+0000 7f500824e640 10 monclient(hunting): authenticate will time out at 1741.690016s 2024-03-25T14:30:56.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.597+0000 7f5005fea640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f50000a98f0 0x7f50001350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:57992/0 (socket says 172.21.15.26:57992) 2024-03-25T14:30:56.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.597+0000 7f5005fea640 1 -- 172.21.15.26:0/2552873212 learned_addr learned my addr 172.21.15.26:0/2552873212 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:30:56.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.600+0000 7f4ff67fc640 10 client.?.objecter ms_handle_connect 0x7f50000a98f0 2024-03-25T14:30:56.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.600+0000 7f4ff67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:56.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.600+0000 7f4ff67fc640 10 client.?.objecter ms_handle_connect 0x7f500013f710 2024-03-25T14:30:56.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.600+0000 7f4ff67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:56.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.605+0000 7f4ff67fc640 10 client.?.objecter ms_handle_connect 0x7f50001430c0 2024-03-25T14:30:56.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.605+0000 7f4ff67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:56.613 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.609+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836134332 0 0) 0x7f50000ab8b0 con 0x7f500013f710 2024-03-25T14:30:56.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.609+0000 7f4ff67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.609+0000 7f4ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.609+0000 7f4ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.609+0000 7f4ff67fc640 10 monclient(hunting): my global_id is 4401 2024-03-25T14:30:56.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.609+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe4004eb0 con 0x7f500013f710 2024-03-25T14:30:56.615 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.611+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244283148 0 0) 0x7f50000abb90 con 0x7f50000a98f0 2024-03-25T14:30:56.615 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.611+0000 7f4ff67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.615 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.611+0000 7f4ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.615 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.611+0000 7f4ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.615 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.611+0000 7f4ff67fc640 10 monclient(hunting): my global_id is 4390 2024-03-25T14:30:56.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.611+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50000ab8b0 con 0x7f50000a98f0 2024-03-25T14:30:56.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336357661 0 0) 0x7f50000ab8b0 con 0x7f50000a98f0 2024-03-25T14:30:56.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50000abb90 con 0x7f50000a98f0 2024-03-25T14:30:56.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ffc0019f0 con 0x7f50000a98f0 2024-03-25T14:30:56.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:56.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 10 monclient(hunting): dump: 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.622+0000 7f4ff67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:56.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.627+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 726651244 0 0) 0x7f4fe4004eb0 con 0x7f500013f710 2024-03-25T14:30:56.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.627+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50000ab8b0 con 0x7f500013f710 2024-03-25T14:30:56.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.627+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ff0001b00 con 0x7f500013f710 2024-03-25T14:30:56.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.627+0000 7f4ff67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.627+0000 7f4ff67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.627+0000 7f4ff67fc640 10 monclient(hunting): dump: 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:56.633 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.628+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500160090 0 0) 0x7f5000137870 con 0x7f50001430c0 2024-03-25T14:30:56.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.628+0000 7f4ff67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.628+0000 7f4ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.628+0000 7f4ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.628+0000 7f4ff67fc640 10 monclient(hunting): my global_id is 4364 2024-03-25T14:30:56.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.628+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe4004eb0 con 0x7f50001430c0 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3864429299 0 0) 0x7f50000abb90 con 0x7f50000a98f0 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 >> v1:172.21.15.26:6790/0 conn(0x7f50001430c0 legacy=0x7f50001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 >> v1:172.21.15.26:6789/0 conn(0x7f500013f710 legacy=0x7f50001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 1 monclient: found mon.b 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50001389e0 con 0x7f50000a98f0 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 10 monclient: _finish_auth 0 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f4ff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f500824e640 5 monclient: authenticate success, global_id 4390 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f500824e640 10 monclient: _renew_subs 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f500824e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.640+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5000136d90 con 0x7f50000a98f0 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.641+0000 7f500824e640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.641+0000 7f500824e640 10 monclient: _renew_subs 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.641+0000 7f500824e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.641+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50001384c0 con 0x7f50000a98f0 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.641+0000 7f500824e640 1 librados: init done 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.642+0000 7f500824e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.642+0000 7f500824e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:56.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.642+0000 7f500824e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:56.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.642+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5000136f80 con 0x7f50000a98f0 2024-03-25T14:30:56.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.933+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ffc003610 con 0x7f50000a98f0 2024-03-25T14:30:56.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.933+0000 7f4ff67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.933+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ffc003a20 con 0x7f50000a98f0 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.933+0000 7f4ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.934+0000 7f4ff67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.934+0000 7f4ff67fc640 10 monclient: dump: 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:30:56.939 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:30:56.940 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:30:56.940 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:30:56.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.933+0000 7f4ff7fff640 4 set_mon_vals no callback set 2024-03-25T14:30:56.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.934+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4ffc003c00 con 0x7f50000a98f0 2024-03-25T14:30:56.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.935+0000 7f50057e9640 10 monclient: get_auth_request con 0x7f4fe406dee0 auth_method 0 2024-03-25T14:30:56.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.935+0000 7f4ff67fc640 10 client.4390.objecter ms_handle_connect 0x7f4fe406dee0 2024-03-25T14:30:56.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f4ffc054840 con 0x7f50000a98f0 2024-03-25T14:30:56.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 10 client.4390.objecter ms_dispatch 0x7f5000000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:30:56.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:30:56.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:30:56.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T14:30:56.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ffc050350 con 0x7f50000a98f0 2024-03-25T14:30:56.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:56.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:56.940+0000 7f4ff67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:57.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.185+0000 7f500824e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:57.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.185+0000 7f500824e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:57.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.185+0000 7f500824e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:30:57.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.185+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5000138b70 con 0x7f50000a98f0 2024-03-25T14:30:57.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.210+0000 7f4ff67fc640 1 -- 172.21.15.26:0/2552873212 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f4ffc052280 con 0x7f50000a98f0 2024-03-25T14:30:57.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.210+0000 7f4ff67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:57.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.210+0000 7f4ff67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:57.215 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:30:57.215 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 10 librados: watch_flush enter 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 10 librados: watch_flush exit 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 10 client.4390.objecter successfully canceled tick 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 >> v1:172.21.15.121:6800/288813066 conn(0x7f4fe406dee0 legacy=0x7f4fe40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 10 monclient: shutdown 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.212+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 >> v1:172.21.15.121:6789/0 conn(0x7f50000a98f0 legacy=0x7f50001350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.213+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 shutdown_connections 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.213+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 >> 172.21.15.26:0/2552873212 conn(0x7f50000b7ea0 msgr2=0x7f5000074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:57.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.213+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 shutdown_connections 2024-03-25T14:30:57.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.213+0000 7f500824e640 1 -- 172.21.15.26:0/2552873212 wait complete. 2024-03-25T14:30:57.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:30:57.213+0000 7f500824e640 1 librados: shutdown 2024-03-25T14:30:57.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:04.038 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:04.154 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.150+0000 7f6d9eb04640 10 monclient: get_monmap_and_config 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.150+0000 7f6d9eb04640 10 monclient: build_initial_monmap 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.150+0000 7f6d9eb04640 10 monclient: monmap: 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:04.150922+0000 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:04.150922+0000 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:04.155 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.150+0000 7f6d9eb04640 1 Processor -- start 2024-03-25T14:31:04.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 1 -- start start 2024-03-25T14:31:04.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient: init 2024-03-25T14:31:04.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:04.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:31:04.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient(hunting): picked mon.noname-c con 0x7f6d980a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient(hunting): picked mon.noname-b con 0x7f6d981430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d98142520 con 0x7f6d9813f710 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d980b28a0 con 0x7f6d980a98f0 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d980b2a30 con 0x7f6d981430c0 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:04.158 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.151+0000 7f6d9eb04640 10 monclient(hunting): authenticate will time out at 1749.254519s 2024-03-25T14:31:04.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.159+0000 7f6d9d0a1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f6d981430c0 0x7f6d9814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40588/0 (socket says 172.21.15.26:40588) 2024-03-25T14:31:04.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.159+0000 7f6d9d0a1640 1 -- 172.21.15.26:0/3466638846 learned_addr learned my addr 172.21.15.26:0/3466638846 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199706324 0 0) 0x7f6d980b28a0 con 0x7f6d980a98f0 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 10 monclient(hunting): my global_id is 4393 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88003520 con 0x7f6d980a98f0 2024-03-25T14:31:04.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.176+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944171782 0 0) 0x7f6d980b2a30 con 0x7f6d981430c0 2024-03-25T14:31:04.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.177+0000 7f6d977fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.177+0000 7f6d977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.177+0000 7f6d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.177+0000 7f6d977fe640 10 monclient(hunting): my global_id is 4367 2024-03-25T14:31:04.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.177+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d980b28a0 con 0x7f6d981430c0 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2250050652 0 0) 0x7f6d980b28a0 con 0x7f6d981430c0 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 >> v1:172.21.15.121:6789/0 conn(0x7f6d980a98f0 legacy=0x7f6d980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 >> v1:172.21.15.26:6789/0 conn(0x7f6d9813f710 legacy=0x7f6d981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: _finish_hunting 0 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 1 monclient: found mon.noname-b 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d980b44c0 con 0x7f6d981430c0 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: _finish_auth 0 2024-03-25T14:31:04.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: _check_auth_tickets 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6d900019f0 con 0x7f6d981430c0 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d977fe640 10 monclient: dump: 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:04.193 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d9eb04640 5 monclient: authenticate success, global_id 4367 2024-03-25T14:31:04.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.187+0000 7f6d9eb04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d90001b50 con 0x7f6d981430c0 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d977fe640 1 -- 172.21.15.26:0/3466638846 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6d90001f20 con 0x7f6d981430c0 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d9eb04640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d9eb04640 4 set_mon_vals no callback set 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.202+0000 7f6d977fe640 10 monclient: dump: 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:04.208 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 10 monclient: shutdown 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 1 -- 172.21.15.26:0/3466638846 >> v1:172.21.15.26:6790/0 conn(0x7f6d981430c0 legacy=0x7f6d9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 1 -- 172.21.15.26:0/3466638846 shutdown_connections 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 1 -- 172.21.15.26:0/3466638846 >> 172.21.15.26:0/3466638846 conn(0x7f6d980b7ea0 msgr2=0x7f6d980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 1 -- 172.21.15.26:0/3466638846 shutdown_connections 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.203+0000 7f6d9eb04640 1 -- 172.21.15.26:0/3466638846 wait complete. 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.204+0000 7f6d9eb04640 10 monclient: build_initial_monmap 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.204+0000 7f6d9eb04640 10 monclient: monmap: 2024-03-25T14:31:04.211 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 1 librados: starting msgr at 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 1 librados: starting objecter 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 1 Processor -- start 2024-03-25T14:31:04.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 1 -- start start 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 1 librados: setting wanted keys 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 1 librados: calling monclient init 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.205+0000 7f6d9eb04640 10 monclient: init 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient(hunting): picked mon.noname-a con 0x7f6d980a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9813f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient(hunting): picked mon.noname-c con 0x7f6d981430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d98139590 con 0x7f6d980a98f0 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d98139800 con 0x7f6d9813f710 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d981389f0 con 0x7f6d981430c0 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:04.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.206+0000 7f6d9eb04640 10 monclient(hunting): authenticate will time out at 1749.309059s 2024-03-25T14:31:04.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.221+0000 7f6d97fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6d9813f710 0x7f6d981d72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:49272/0 (socket says 172.21.15.26:49272) 2024-03-25T14:31:04.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.221+0000 7f6d97fff640 1 -- 172.21.15.26:0/4029561777 learned_addr learned my addr 172.21.15.26:0/4029561777 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:04.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.224+0000 7f6d957fa640 10 client.?.objecter ms_handle_connect 0x7f6d9813f710 2024-03-25T14:31:04.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.224+0000 7f6d957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:04.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.227+0000 7f6d957fa640 10 client.?.objecter ms_handle_connect 0x7f6d981430c0 2024-03-25T14:31:04.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.227+0000 7f6d957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:04.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.237+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926325758 0 0) 0x7f6d981389f0 con 0x7f6d981430c0 2024-03-25T14:31:04.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.237+0000 7f6d957fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.237+0000 7f6d957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.237+0000 7f6d957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.237+0000 7f6d957fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T14:31:04.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.237+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d981430c0 2024-03-25T14:31:04.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.239+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928866968 0 0) 0x7f6d98139800 con 0x7f6d9813f710 2024-03-25T14:31:04.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.239+0000 7f6d957fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.239+0000 7f6d957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.239+0000 7f6d957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.239+0000 7f6d957fa640 10 monclient(hunting): my global_id is 4396 2024-03-25T14:31:04.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.239+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d981389f0 con 0x7f6d9813f710 2024-03-25T14:31:04.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.243+0000 7f6d957fa640 10 client.?.objecter ms_handle_connect 0x7f6d980a98f0 2024-03-25T14:31:04.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.243+0000 7f6d957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2044142052 0 0) 0x7f6d7c004eb0 con 0x7f6d981430c0 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d98139800 con 0x7f6d981430c0 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6d90001e60 con 0x7f6d981430c0 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 10 monclient(hunting): dump: 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:04.269 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.263+0000 7f6d957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:04.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.265+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660557094 0 0) 0x7f6d981389f0 con 0x7f6d9813f710 2024-03-25T14:31:04.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.265+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d9813f710 2024-03-25T14:31:04.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.265+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6d84001990 con 0x7f6d9813f710 2024-03-25T14:31:04.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.265+0000 7f6d957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.265+0000 7f6d957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.265+0000 7f6d957fa640 10 monclient(hunting): dump: 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.266+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197776759 0 0) 0x7f6d98139590 con 0x7f6d980a98f0 2024-03-25T14:31:04.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.266+0000 7f6d957fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.266+0000 7f6d957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.266+0000 7f6d957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.266+0000 7f6d957fa640 10 monclient(hunting): my global_id is 4407 2024-03-25T14:31:04.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.266+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d981389f0 con 0x7f6d980a98f0 2024-03-25T14:31:04.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027951856 0 0) 0x7f6d98139800 con 0x7f6d981430c0 2024-03-25T14:31:04.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 >> v1:172.21.15.121:6789/0 conn(0x7f6d9813f710 legacy=0x7f6d981d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 >> v1:172.21.15.26:6789/0 conn(0x7f6d980a98f0 legacy=0x7f6d981376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 10 monclient: _finish_hunting 0 2024-03-25T14:31:04.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 1 monclient: found mon.c 2024-03-25T14:31:04.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d98137f20 con 0x7f6d981430c0 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 10 monclient: _finish_auth 0 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 10 monclient: _check_auth_tickets 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 5 monclient: authenticate success, global_id 4370 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 10 monclient: _renew_subs 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d98138170 con 0x7f6d981430c0 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 10 monclient: _renew_subs 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:04.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d981d7be0 con 0x7f6d981430c0 2024-03-25T14:31:04.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.273+0000 7f6d9eb04640 1 librados: init done 2024-03-25T14:31:04.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.274+0000 7f6d9eb04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:04.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.274+0000 7f6d9eb04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:04.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.274+0000 7f6d9eb04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:04.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.274+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d98138840 con 0x7f6d981430c0 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d900020a0 con 0x7f6d981430c0 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6d90004fb0 con 0x7f6d981430c0 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d957fa640 10 monclient: dump: 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.600 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:04.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.595+0000 7f6d96ffd640 4 set_mon_vals no callback set 2024-03-25T14:31:04.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.596+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f6d9001af80 con 0x7f6d981430c0 2024-03-25T14:31:04.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.596+0000 7f6d9c8a0640 10 monclient: get_auth_request con 0x7f6d7c06deb0 auth_method 0 2024-03-25T14:31:04.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.597+0000 7f6d957fa640 10 client.4370.objecter ms_handle_connect 0x7f6d7c06deb0 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f6d90054bc0 con 0x7f6d981430c0 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 10 client.4370.objecter ms_dispatch 0x7f6d98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d90050500 con 0x7f6d981430c0 2024-03-25T14:31:04.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:04.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.601+0000 7f6d957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:04.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.856+0000 7f6d9eb04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:04.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.856+0000 7f6d9eb04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:04.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.856+0000 7f6d9eb04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:04.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.856+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d98074cc0 con 0x7f6d981430c0 2024-03-25T14:31:04.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.893+0000 7f6d957fa640 1 -- 172.21.15.26:0/4029561777 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f6d90052600 con 0x7f6d981430c0 2024-03-25T14:31:04.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.893+0000 7f6d957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:04.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.893+0000 7f6d957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:04.899 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:04.899 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:04.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 10 librados: watch_flush enter 2024-03-25T14:31:04.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 10 librados: watch_flush exit 2024-03-25T14:31:04.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T14:31:04.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 10 client.4370.objecter successfully canceled tick 2024-03-25T14:31:04.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 >> v1:172.21.15.121:6800/288813066 conn(0x7f6d7c06deb0 legacy=0x7f6d7c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 10 monclient: shutdown 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 >> v1:172.21.15.26:6790/0 conn(0x7f6d981430c0 legacy=0x7f6d98133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 shutdown_connections 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 >> 172.21.15.26:0/4029561777 conn(0x7f6d980b7ea0 msgr2=0x7f6d98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 shutdown_connections 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 -- 172.21.15.26:0/4029561777 wait complete. 2024-03-25T14:31:04.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:04.896+0000 7f6d9eb04640 1 librados: shutdown 2024-03-25T14:31:04.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:11.719 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:11.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.832+0000 7f6e36df4640 10 monclient: get_monmap_and_config 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.832+0000 7f6e36df4640 10 monclient: build_initial_monmap 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.832+0000 7f6e36df4640 10 monclient: monmap: 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:11.832909+0000 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:11.832909+0000 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:11.837 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:11.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 1 Processor -- start 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 1 -- start start 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient: init 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient(hunting): picked mon.noname-c con 0x7f6e300a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient(hunting): picked mon.noname-b con 0x7f6e3013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient(hunting): picked mon.noname-a con 0x7f6e301430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e30142520 con 0x7f6e301430c0 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300b28a0 con 0x7f6e300a98f0 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300b2a30 con 0x7f6e3013f710 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:11.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.833+0000 7f6e36df4640 10 monclient(hunting): authenticate will time out at 1756.936522s 2024-03-25T14:31:11.840 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.835+0000 7f6e34b90640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6e300a98f0 0x7f6e300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:47610/0 (socket says 172.21.15.26:47610) 2024-03-25T14:31:11.840 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.835+0000 7f6e34b90640 1 -- 172.21.15.26:0/3137089170 learned_addr learned my addr 172.21.15.26:0/3137089170 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:11.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.844+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329866061 0 0) 0x7f6e300b28a0 con 0x7f6e300a98f0 2024-03-25T14:31:11.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.844+0000 7f6e2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.844+0000 7f6e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.844+0000 7f6e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.844+0000 7f6e2f7fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T14:31:11.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.844+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e18003520 con 0x7f6e300a98f0 2024-03-25T14:31:11.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.853+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174802225 0 0) 0x7f6e300b2a30 con 0x7f6e3013f710 2024-03-25T14:31:11.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.853+0000 7f6e2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.853+0000 7f6e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.853+0000 7f6e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.853+0000 7f6e2f7fe640 10 monclient(hunting): my global_id is 4373 2024-03-25T14:31:11.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.853+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e300b28a0 con 0x7f6e3013f710 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 337685897 0 0) 0x7f6e18003520 con 0x7f6e300a98f0 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 >> v1:172.21.15.26:6790/0 conn(0x7f6e3013f710 legacy=0x7f6e301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 >> v1:172.21.15.26:6789/0 conn(0x7f6e301430c0 legacy=0x7f6e3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 1 monclient: found mon.noname-c 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:11.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e300b44c0 con 0x7f6e300a98f0 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6e28001690 con 0x7f6e300a98f0 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e2f7fe640 10 monclient: dump: 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.860 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:11.861 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:11.861 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:11.861 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:11.861 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:11.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e36df4640 5 monclient: authenticate success, global_id 4399 2024-03-25T14:31:11.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.854+0000 7f6e36df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:11.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.864+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e280017f0 con 0x7f6e300a98f0 2024-03-25T14:31:11.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.864+0000 7f6e2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:11.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.864+0000 7f6e2f7fe640 1 -- 172.21.15.26:0/3137089170 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6e28001c00 con 0x7f6e300a98f0 2024-03-25T14:31:11.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.864+0000 7f6e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.864+0000 7f6e2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:11.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.864+0000 7f6e2f7fe640 10 monclient: dump: 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.865+0000 7f6e36df4640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:11.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.865+0000 7f6e36df4640 4 set_mon_vals no callback set 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.865+0000 7f6e36df4640 10 monclient: shutdown 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.865+0000 7f6e36df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.865+0000 7f6e36df4640 1 -- 172.21.15.26:0/3137089170 >> v1:172.21.15.121:6789/0 conn(0x7f6e300a98f0 legacy=0x7f6e300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.866+0000 7f6e36df4640 1 -- 172.21.15.26:0/3137089170 shutdown_connections 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.866+0000 7f6e36df4640 1 -- 172.21.15.26:0/3137089170 >> 172.21.15.26:0/3137089170 conn(0x7f6e300b7ea0 msgr2=0x7f6e300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.866+0000 7f6e36df4640 1 -- 172.21.15.26:0/3137089170 shutdown_connections 2024-03-25T14:31:11.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.866+0000 7f6e36df4640 1 -- 172.21.15.26:0/3137089170 wait complete. 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 10 monclient: build_initial_monmap 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 10 monclient: monmap: 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 1 librados: starting msgr at 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 1 librados: starting objecter 2024-03-25T14:31:11.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 1 Processor -- start 2024-03-25T14:31:11.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 1 -- start start 2024-03-25T14:31:11.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 1 librados: setting wanted keys 2024-03-25T14:31:11.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 1 librados: calling monclient init 2024-03-25T14:31:11.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.867+0000 7f6e36df4640 10 monclient: init 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.868+0000 7f6e36df4640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.868+0000 7f6e36df4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.868+0000 7f6e36df4640 10 monclient(hunting): picked mon.noname-b con 0x7f6e300a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 10 monclient(hunting): picked mon.noname-c con 0x7f6e3013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 10 monclient(hunting): picked mon.noname-a con 0x7f6e301430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e30135730 con 0x7f6e301430c0 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e301358c0 con 0x7f6e300a98f0 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e30135a50 con 0x7f6e3013f710 2024-03-25T14:31:11.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:11.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.869+0000 7f6e36df4640 10 monclient(hunting): authenticate will time out at 1756.971752s 2024-03-25T14:31:11.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.877+0000 7f6e35391640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e301430c0 0x7f6e301d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55096/0 (socket says 172.21.15.26:55096) 2024-03-25T14:31:11.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.877+0000 7f6e35391640 1 -- 172.21.15.26:0/3261528225 learned_addr learned my addr 172.21.15.26:0/3261528225 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:11.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.879+0000 7f6e2d7fa640 10 client.?.objecter ms_handle_connect 0x7f6e301430c0 2024-03-25T14:31:11.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.879+0000 7f6e2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:11.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.882+0000 7f6e2d7fa640 10 client.?.objecter ms_handle_connect 0x7f6e300a98f0 2024-03-25T14:31:11.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.882+0000 7f6e2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:11.893 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 10 client.?.objecter ms_handle_connect 0x7f6e3013f710 2024-03-25T14:31:11.893 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:11.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489471685 0 0) 0x7f6e301358c0 con 0x7f6e300a98f0 2024-03-25T14:31:11.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T14:31:11.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.889+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c004eb0 con 0x7f6e300a98f0 2024-03-25T14:31:11.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.896+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693313706 0 0) 0x7f6e30135730 con 0x7f6e301430c0 2024-03-25T14:31:11.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.896+0000 7f6e2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.896+0000 7f6e2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.896+0000 7f6e2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.896+0000 7f6e2d7fa640 10 monclient(hunting): my global_id is 4413 2024-03-25T14:31:11.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.896+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e301358c0 con 0x7f6e301430c0 2024-03-25T14:31:11.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.901+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224743194 0 0) 0x7f6e1c004eb0 con 0x7f6e300a98f0 2024-03-25T14:31:11.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.901+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e1c005490 con 0x7f6e300a98f0 2024-03-25T14:31:11.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.903+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368223512 0 0) 0x7f6e30135a50 con 0x7f6e3013f710 2024-03-25T14:31:11.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.903+0000 7f6e2d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.903+0000 7f6e2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.903+0000 7f6e2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.903+0000 7f6e2d7fa640 10 monclient(hunting): my global_id is 4376 2024-03-25T14:31:11.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.903+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c004eb0 con 0x7f6e3013f710 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6e28001be0 con 0x7f6e300a98f0 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:11.917 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386484333 0 0) 0x7f6e1c005490 con 0x7f6e300a98f0 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 >> v1:172.21.15.26:6790/0 conn(0x7f6e3013f710 legacy=0x7f6e301d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 >> v1:172.21.15.26:6789/0 conn(0x7f6e301430c0 legacy=0x7f6e301d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 1 monclient: found mon.b 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e301d5000 con 0x7f6e300a98f0 2024-03-25T14:31:11.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 5 monclient: authenticate success, global_id 4402 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 10 monclient: _renew_subs 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e301d52b0 con 0x7f6e300a98f0 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 10 monclient: _renew_subs 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3013c900 con 0x7f6e300a98f0 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.911+0000 7f6e36df4640 1 librados: init done 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.912+0000 7f6e36df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:11.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.912+0000 7f6e36df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:11.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.912+0000 7f6e36df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:11.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.912+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e301371c0 con 0x7f6e300a98f0 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e28004a90 con 0x7f6e300a98f0 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6e28004da0 con 0x7f6e300a98f0 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2d7fa640 10 monclient: dump: 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.934 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:11.935 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:11.935 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:11.935 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:11.935 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:11.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.929+0000 7f6e2effd640 4 set_mon_vals no callback set 2024-03-25T14:31:11.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.930+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f6e2801ad70 con 0x7f6e300a98f0 2024-03-25T14:31:11.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.930+0000 7f6e2ffff640 10 monclient: get_auth_request con 0x7f6e1c06dfe0 auth_method 0 2024-03-25T14:31:11.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.930+0000 7f6e2d7fa640 10 client.4402.objecter ms_handle_connect 0x7f6e1c06dfe0 2024-03-25T14:31:11.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.933+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f6e28054930 con 0x7f6e300a98f0 2024-03-25T14:31:11.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.933+0000 7f6e2d7fa640 10 client.4402.objecter ms_dispatch 0x7f6e30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:11.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.933+0000 7f6e2d7fa640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:11.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.933+0000 7f6e2d7fa640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:11.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:11.933+0000 7f6e2d7fa640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T14:31:12.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.190+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e28050270 con 0x7f6e300a98f0 2024-03-25T14:31:12.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.191+0000 7f6e2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:12.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.191+0000 7f6e2d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:12.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.450+0000 7f6e36df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:12.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.450+0000 7f6e36df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:12.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.450+0000 7f6e36df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:12.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.450+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e300627b0 con 0x7f6e300a98f0 2024-03-25T14:31:12.474 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.470+0000 7f6e2d7fa640 1 -- 172.21.15.26:0/3261528225 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f6e28052370 con 0x7f6e300a98f0 2024-03-25T14:31:12.474 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.470+0000 7f6e2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:12.475 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.470+0000 7f6e2d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:12.475 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:12.475 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 10 librados: watch_flush enter 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 10 librados: watch_flush exit 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 10 client.4402.objecter successfully canceled tick 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 >> v1:172.21.15.121:6800/288813066 conn(0x7f6e1c06dfe0 legacy=0x7f6e1c0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 10 monclient: shutdown 2024-03-25T14:31:12.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:12.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 >> v1:172.21.15.121:6789/0 conn(0x7f6e300a98f0 legacy=0x7f6e301dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:12.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 shutdown_connections 2024-03-25T14:31:12.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 >> 172.21.15.26:0/3261528225 conn(0x7f6e300b7ea0 msgr2=0x7f6e3014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:12.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 shutdown_connections 2024-03-25T14:31:12.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 -- 172.21.15.26:0/3261528225 wait complete. 2024-03-25T14:31:12.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:12.472+0000 7f6e36df4640 1 librados: shutdown 2024-03-25T14:31:12.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:19.296 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:19.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.391+0000 7f4dc84c1640 10 monclient: get_monmap_and_config 2024-03-25T14:31:19.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.391+0000 7f4dc84c1640 10 monclient: build_initial_monmap 2024-03-25T14:31:19.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.391+0000 7f4dc84c1640 10 monclient: monmap: 2024-03-25T14:31:19.396 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:19.396 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:19.392074+0000 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:19.392074+0000 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.391+0000 7f4dc84c1640 1 Processor -- start 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.391+0000 7f4dc84c1640 1 -- start start 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.391+0000 7f4dc84c1640 10 monclient: init 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc00b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc00b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc014be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00af940 con 0x7f4dc00b4d60 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00b0a70 con 0x7f4dc00b1730 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00b0d50 con 0x7f4dc014be90 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.392+0000 7f4dc84c1640 10 monclient(hunting): authenticate will time out at 1764.495277s 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.393+0000 7f4dc6a5e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4dc014be90 0x7f4dc014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:34606/0 (socket says 172.21.15.26:34606) 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.393+0000 7f4dc6a5e640 1 -- 172.21.15.26:0/1433498007 learned_addr learned my addr 172.21.15.26:0/1433498007 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.426+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830200422 0 0) 0x7f4dc00af940 con 0x7f4dc00b4d60 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.426+0000 7f4dc525b640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.426+0000 7f4dc525b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.426+0000 7f4dc525b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.426+0000 7f4dc525b640 10 monclient(hunting): my global_id is 4416 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.426+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac003520 con 0x7f4dc00b4d60 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772188002 0 0) 0x7f4dac003520 con 0x7f4dc00b4d60 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 >> v1:172.21.15.26:6790/0 conn(0x7f4dc014be90 legacy=0x7f4dc014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 >> v1:172.21.15.121:6789/0 conn(0x7f4dc00b1730 legacy=0x7f4dc00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: _finish_hunting 0 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 1 monclient: found mon.noname-a 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc00b1030 con 0x7f4dc00b4d60 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: _finish_auth 0 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: _check_auth_tickets 2024-03-25T14:31:19.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4db0001690 con 0x7f4dc00b4d60 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc84c1640 5 monclient: authenticate success, global_id 4416 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc84c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc525b640 10 monclient: dump: 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:19.453 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:19.454 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:19.454 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:19.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.446+0000 7f4dc84c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4db00017f0 con 0x7f4dc00b4d60 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc525b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc525b640 1 -- 172.21.15.26:0/1433498007 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4db0001c00 con 0x7f4dc00b4d60 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc525b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc525b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc525b640 10 monclient: dump: 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.473 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc84c1640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:19.474 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.468+0000 7f4dc84c1640 4 set_mon_vals no callback set 2024-03-25T14:31:19.475 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 10 monclient: shutdown 2024-03-25T14:31:19.475 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 1 -- 172.21.15.26:0/1433498007 >> v1:172.21.15.26:6789/0 conn(0x7f4dc00b4d60 legacy=0x7f4dc013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 1 -- 172.21.15.26:0/1433498007 shutdown_connections 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 1 -- 172.21.15.26:0/1433498007 >> 172.21.15.26:0/1433498007 conn(0x7f4dc00617a0 msgr2=0x7f4dc0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 1 -- 172.21.15.26:0/1433498007 shutdown_connections 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.469+0000 7f4dc84c1640 1 -- 172.21.15.26:0/1433498007 wait complete. 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.470+0000 7f4dc84c1640 10 monclient: build_initial_monmap 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.470+0000 7f4dc84c1640 10 monclient: monmap: 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:19.476 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.470+0000 7f4dc84c1640 1 librados: starting msgr at 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.470+0000 7f4dc84c1640 1 librados: starting objecter 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 1 Processor -- start 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 1 -- start start 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 1 librados: setting wanted keys 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 1 librados: calling monclient init 2024-03-25T14:31:19.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 10 monclient: init 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc00b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.471+0000 7f4dc84c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc00b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.472+0000 7f4dc84c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc014be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.472+0000 7f4dc84c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc0142f70 con 0x7f4dc014be90 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.472+0000 7f4dc84c1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc01411b0 con 0x7f4dc00b4d60 2024-03-25T14:31:19.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.472+0000 7f4dc84c1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc0143250 con 0x7f4dc00b1730 2024-03-25T14:31:19.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.472+0000 7f4dc84c1640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:19.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.472+0000 7f4dc84c1640 10 monclient(hunting): authenticate will time out at 1764.574651s 2024-03-25T14:31:19.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.478+0000 7f4dc6a5e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4dc014be90 0x7f4dc0141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55318/0 (socket says 172.21.15.26:55318) 2024-03-25T14:31:19.483 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.478+0000 7f4dc6a5e640 1 -- 172.21.15.26:0/2219922255 learned_addr learned my addr 172.21.15.26:0/2219922255 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:19.487 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.482+0000 7f4db6ffd640 10 client.?.objecter ms_handle_connect 0x7f4dc014be90 2024-03-25T14:31:19.487 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.482+0000 7f4db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:19.488 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.484+0000 7f4db6ffd640 10 client.?.objecter ms_handle_connect 0x7f4dc00b4d60 2024-03-25T14:31:19.488 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.484+0000 7f4db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:19.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.492+0000 7f4db6ffd640 10 client.?.objecter ms_handle_connect 0x7f4dc00b1730 2024-03-25T14:31:19.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.492+0000 7f4db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:19.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.499+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699191019 0 0) 0x7f4dc0142f70 con 0x7f4dc014be90 2024-03-25T14:31:19.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.499+0000 7f4db6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.499+0000 7f4db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.499+0000 7f4db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.499+0000 7f4db6ffd640 10 monclient(hunting): my global_id is 4419 2024-03-25T14:31:19.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.499+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da4004eb0 con 0x7f4dc014be90 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3402242973 0 0) 0x7f4da4004eb0 con 0x7f4dc014be90 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dc0142f70 con 0x7f4dc014be90 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4db8001b10 con 0x7f4dc014be90 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 10 monclient(hunting): dump: 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.577 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.578 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:19.578 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:19.578 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:19.578 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:19.578 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:19.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.572+0000 7f4db6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:19.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.582+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934176824 0 0) 0x7f4dc0143250 con 0x7f4dc00b1730 2024-03-25T14:31:19.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.582+0000 7f4db6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.582+0000 7f4db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.582+0000 7f4db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.582+0000 7f4db6ffd640 10 monclient(hunting): my global_id is 4382 2024-03-25T14:31:19.587 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.582+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da4004eb0 con 0x7f4dc00b1730 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1453938020 0 0) 0x7f4dc0142f70 con 0x7f4dc014be90 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 >> v1:172.21.15.26:6790/0 conn(0x7f4dc00b1730 legacy=0x7f4dc01372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 >> v1:172.21.15.121:6789/0 conn(0x7f4dc00b4d60 legacy=0x7f4dc013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 1 monclient: found mon.a 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:19.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc0140c90 con 0x7f4dc014be90 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 5 monclient: authenticate success, global_id 4419 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 10 monclient: _renew_subs 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc01379c0 con 0x7f4dc014be90 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 10 monclient: _renew_subs 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc013d9e0 con 0x7f4dc014be90 2024-03-25T14:31:19.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.605+0000 7f4dc84c1640 1 librados: init done 2024-03-25T14:31:19.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.606+0000 7f4dc84c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:19.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.606+0000 7f4dc84c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:19.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.606+0000 7f4dc84c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:31:19.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.606+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc00627b0 con 0x7f4dc014be90 2024-03-25T14:31:19.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.974+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4db8002040 con 0x7f4dc014be90 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.975+0000 7f4db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.975+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4db8004d00 con 0x7f4dc014be90 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.975+0000 7f4db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.975+0000 7f4db6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.975+0000 7f4db6ffd640 10 monclient: dump: 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:19.980 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:19.981 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:19.981 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:19.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.975+0000 7f4dc4a5a640 4 set_mon_vals no callback set 2024-03-25T14:31:19.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.976+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4db8004f20 con 0x7f4dc014be90 2024-03-25T14:31:19.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.976+0000 7f4dc625d640 10 monclient: get_auth_request con 0x7f4da406dd40 auth_method 0 2024-03-25T14:31:19.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.977+0000 7f4db6ffd640 10 client.4419.objecter ms_handle_connect 0x7f4da406dd40 2024-03-25T14:31:19.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f4db8054990 con 0x7f4dc014be90 2024-03-25T14:31:19.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 10 client.4419.objecter ms_dispatch 0x7f4dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:19.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:19.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:19.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-25T14:31:19.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4db80504a0 con 0x7f4dc014be90 2024-03-25T14:31:19.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:19.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:19.981+0000 7f4db6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:20.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.244+0000 7f4dc84c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:20.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.244+0000 7f4dc84c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:20.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.244+0000 7f4dc84c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:31:20.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.244+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dc0140e20 con 0x7f4dc014be90 2024-03-25T14:31:20.290 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.285+0000 7f4db6ffd640 1 -- 172.21.15.26:0/2219922255 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f4db80523d0 con 0x7f4dc014be90 2024-03-25T14:31:20.290 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.285+0000 7f4db6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:20.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.285+0000 7f4db6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:20.291 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:20.291 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 10 librados: watch_flush enter 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 10 librados: watch_flush exit 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 10 client.4419.objecter successfully canceled tick 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 >> v1:172.21.15.121:6800/288813066 conn(0x7f4da406dd40 legacy=0x7f4da4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 10 monclient: shutdown 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 >> v1:172.21.15.26:6789/0 conn(0x7f4dc014be90 legacy=0x7f4dc0141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 shutdown_connections 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 >> 172.21.15.26:0/2219922255 conn(0x7f4dc00617a0 msgr2=0x7f4dc00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:20.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 shutdown_connections 2024-03-25T14:31:20.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 -- 172.21.15.26:0/2219922255 wait complete. 2024-03-25T14:31:20.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:20.288+0000 7f4dc84c1640 1 librados: shutdown 2024-03-25T14:31:20.304 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:27.113 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.223+0000 7f39a2124640 10 monclient: get_monmap_and_config 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.223+0000 7f39a2124640 10 monclient: build_initial_monmap 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.223+0000 7f39a2124640 10 monclient: monmap: 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:27.224473+0000 2024-03-25T14:31:27.228 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:27.224473+0000 2024-03-25T14:31:27.229 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:27.229 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.229 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:27.229 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:27.229 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:27.229 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.224+0000 7f39a2124640 1 Processor -- start 2024-03-25T14:31:27.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.224+0000 7f39a2124640 1 -- start start 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.224+0000 7f39a2124640 10 monclient: init 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient(hunting): picked mon.noname-a con 0x7f399c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient(hunting): picked mon.noname-b con 0x7f399c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient(hunting): picked mon.noname-c con 0x7f399c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c142520 con 0x7f399c0a98f0 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0b28a0 con 0x7f399c1430c0 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0b2a30 con 0x7f399c13f710 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:27.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.225+0000 7f39a2124640 10 monclient(hunting): authenticate will time out at 1772.328054s 2024-03-25T14:31:27.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.232+0000 7f399b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f399c0a98f0 0x7f399c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60016/0 (socket says 172.21.15.26:60016) 2024-03-25T14:31:27.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.232+0000 7f399b7fe640 1 -- 172.21.15.26:0/2439318357 learned_addr learned my addr 172.21.15.26:0/2439318357 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:27.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.252+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625244936 0 0) 0x7f399c142520 con 0x7f399c0a98f0 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.252+0000 7f399a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.252+0000 7f399a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.252+0000 7f399a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.252+0000 7f399a7fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.252+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3984003520 con 0x7f399c0a98f0 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.253+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033184974 0 0) 0x7f399c0b28a0 con 0x7f399c1430c0 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.253+0000 7f399a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.253+0000 7f399a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.253+0000 7f399a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.253+0000 7f399a7fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T14:31:27.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.253+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c142520 con 0x7f399c1430c0 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.266+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057360297 0 0) 0x7f399c142520 con 0x7f399c1430c0 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 >> v1:172.21.15.26:6790/0 conn(0x7f399c13f710 legacy=0x7f399c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 >> v1:172.21.15.26:6789/0 conn(0x7f399c0a98f0 legacy=0x7f399c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 1 monclient: found mon.noname-c 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0b44c0 con 0x7f399c1430c0 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:31:27.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f39940019f0 con 0x7f399c1430c0 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f39a2124640 5 monclient: authenticate success, global_id 4411 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f39a2124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f399a7fc640 10 monclient: dump: 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.273 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:27.274 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:27.274 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:27.274 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.267+0000 7f39a2124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:27.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3994001b50 con 0x7f399c1430c0 2024-03-25T14:31:27.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f399a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:27.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f399a7fc640 1 -- 172.21.15.26:0/2439318357 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3994001f20 con 0x7f399c1430c0 2024-03-25T14:31:27.291 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 4 set_mon_vals no callback set 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f399a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f399a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f399a7fc640 10 monclient: dump: 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:27.292 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 10 monclient: shutdown 2024-03-25T14:31:27.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:27.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 1 -- 172.21.15.26:0/2439318357 >> v1:172.21.15.121:6789/0 conn(0x7f399c1430c0 legacy=0x7f399c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 1 -- 172.21.15.26:0/2439318357 shutdown_connections 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.287+0000 7f39a2124640 1 -- 172.21.15.26:0/2439318357 >> 172.21.15.26:0/2439318357 conn(0x7f399c0b7ea0 msgr2=0x7f399c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.288+0000 7f39a2124640 1 -- 172.21.15.26:0/2439318357 shutdown_connections 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.288+0000 7f39a2124640 1 -- 172.21.15.26:0/2439318357 wait complete. 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.288+0000 7f39a2124640 10 monclient: build_initial_monmap 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.288+0000 7f39a2124640 10 monclient: monmap: 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:27.296 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 1 librados: starting msgr at 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 1 librados: starting objecter 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 1 Processor -- start 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 1 -- start start 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 1 librados: setting wanted keys 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 1 librados: calling monclient init 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 10 monclient: init 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:31:27.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.289+0000 7f39a2124640 10 monclient(hunting): picked mon.noname-c con 0x7f399c0a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 10 monclient(hunting): picked mon.noname-a con 0x7f399c13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 10 monclient(hunting): picked mon.noname-b con 0x7f399c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c138600 con 0x7f399c13f710 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c0b1620 con 0x7f399c1430c0 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399c1346b0 con 0x7f399c0a98f0 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:27.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.290+0000 7f39a2124640 10 monclient(hunting): authenticate will time out at 1772.392783s 2024-03-25T14:31:27.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.306+0000 7f399b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f399c0a98f0 0x7f399c137690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39404/0 (socket says 172.21.15.26:39404) 2024-03-25T14:31:27.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.306+0000 7f399b7fe640 1 -- 172.21.15.26:0/1595933086 learned_addr learned my addr 172.21.15.26:0/1595933086 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:27.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.307+0000 7f3983fff640 10 client.?.objecter ms_handle_connect 0x7f399c0a98f0 2024-03-25T14:31:27.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.307+0000 7f3983fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:27.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.313+0000 7f3983fff640 10 client.?.objecter ms_handle_connect 0x7f399c13f710 2024-03-25T14:31:27.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.313+0000 7f3983fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:27.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.316+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687404671 0 0) 0x7f399c1346b0 con 0x7f399c0a98f0 2024-03-25T14:31:27.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 10 monclient(hunting): _negotiate 2024-03-25T14:31:27.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:27.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:27.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 10 monclient(hunting): my global_id is 4385 2024-03-25T14:31:27.322 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3988004eb0 con 0x7f399c0a98f0 2024-03-25T14:31:27.322 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 10 client.?.objecter ms_handle_connect 0x7f399c1430c0 2024-03-25T14:31:27.322 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.317+0000 7f3983fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:27.330 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687453698 0 0) 0x7f399c138600 con 0x7f399c13f710 2024-03-25T14:31:27.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): _negotiate 2024-03-25T14:31:27.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:27.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:27.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): my global_id is 4425 2024-03-25T14:31:27.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c1346b0 con 0x7f399c13f710 2024-03-25T14:31:27.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413515710 0 0) 0x7f399c0b1620 con 0x7f399c1430c0 2024-03-25T14:31:27.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): _negotiate 2024-03-25T14:31:27.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:27.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:27.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 10 monclient(hunting): my global_id is 4414 2024-03-25T14:31:27.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.326+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c138600 con 0x7f399c1430c0 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557253163 0 0) 0x7f3988004eb0 con 0x7f399c0a98f0 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f399c0b1620 con 0x7f399c0a98f0 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f398c001950 con 0x7f399c0a98f0 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 10 monclient(hunting): dump: 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:27.341 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.342 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.342 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.342 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:27.342 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.342 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:27.342 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:27.343 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:27.343 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.336+0000 7f3983fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:27.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.339+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 88342326 0 0) 0x7f399c138600 con 0x7f399c1430c0 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.339+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3988004eb0 con 0x7f399c1430c0 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.339+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3994001e60 con 0x7f399c1430c0 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.339+0000 7f3983fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.339+0000 7f3983fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.339+0000 7f3983fff640 10 monclient(hunting): dump: 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:27.345 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.340+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434924182 0 0) 0x7f399c1346b0 con 0x7f399c13f710 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.340+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f399c138600 con 0x7f399c13f710 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.340+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f39900018b0 con 0x7f399c13f710 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.340+0000 7f3983fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.340+0000 7f3983fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.340+0000 7f3983fff640 10 monclient(hunting): dump: 2024-03-25T14:31:27.346 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:27.347 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.347 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.347 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.347 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:27.347 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.347 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:27.348 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:27.348 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:27.348 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.351+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801737564 0 0) 0x7f3988004eb0 con 0x7f399c1430c0 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 >> v1:172.21.15.26:6790/0 conn(0x7f399c0a98f0 legacy=0x7f399c137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 >> v1:172.21.15.26:6789/0 conn(0x7f399c13f710 legacy=0x7f399c133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 10 monclient: _finish_hunting 0 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 1 monclient: found mon.b 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c134840 con 0x7f399c1430c0 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 10 monclient: _finish_auth 0 2024-03-25T14:31:27.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 10 monclient: _check_auth_tickets 2024-03-25T14:31:27.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f3983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:27.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 5 monclient: authenticate success, global_id 4414 2024-03-25T14:31:27.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 10 monclient: _renew_subs 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c137f50 con 0x7f399c1430c0 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 10 monclient: _renew_subs 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c1daf90 con 0x7f399c1430c0 2024-03-25T14:31:27.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.352+0000 7f39a2124640 1 librados: init done 2024-03-25T14:31:27.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.353+0000 7f39a2124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:27.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.353+0000 7f39a2124640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:27.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.353+0000 7f39a2124640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:27.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.353+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c138430 con 0x7f399c1430c0 2024-03-25T14:31:27.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.356+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3994001fe0 con 0x7f399c1430c0 2024-03-25T14:31:27.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.356+0000 7f3983fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:27.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.356+0000 7f3999ffb640 4 set_mon_vals no callback set 2024-03-25T14:31:27.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.628+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3994003dc0 con 0x7f399c1430c0 2024-03-25T14:31:27.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.628+0000 7f3983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:27.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.628+0000 7f3983fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:27.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.628+0000 7f3983fff640 10 monclient: dump: 2024-03-25T14:31:27.633 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:27.633 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:27.634 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:27.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.629+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f3994019d90 con 0x7f399c1430c0 2024-03-25T14:31:27.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.630+0000 7f399b7fe640 10 monclient: get_auth_request con 0x7f398806e160 auth_method 0 2024-03-25T14:31:27.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.630+0000 7f3983fff640 10 client.4414.objecter ms_handle_connect 0x7f398806e160 2024-03-25T14:31:27.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.632+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f39940538b0 con 0x7f399c1430c0 2024-03-25T14:31:27.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.632+0000 7f3983fff640 10 client.4414.objecter ms_dispatch 0x7f399c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:27.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.632+0000 7f3983fff640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:27.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.632+0000 7f3983fff640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:27.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.632+0000 7f3983fff640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T14:31:27.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.637+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f399404f3c0 con 0x7f399c1430c0 2024-03-25T14:31:27.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.637+0000 7f3983fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:27.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.637+0000 7f3983fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:27.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.887+0000 7f39a2124640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:27.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.887+0000 7f39a2124640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:27.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.887+0000 7f39a2124640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:27.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.887+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f399c14f300 con 0x7f399c1430c0 2024-03-25T14:31:27.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.916+0000 7f3983fff640 1 -- 172.21.15.26:0/1595933086 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f39940512f0 con 0x7f399c1430c0 2024-03-25T14:31:27.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.916+0000 7f3983fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:27.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.916+0000 7f3983fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:27.921 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:27.921 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 10 librados: watch_flush enter 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 10 librados: watch_flush exit 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 10 client.4414.objecter successfully canceled tick 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 >> v1:172.21.15.121:6800/288813066 conn(0x7f398806e160 legacy=0x7f3988070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 10 monclient: shutdown 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 >> v1:172.21.15.121:6789/0 conn(0x7f399c1430c0 legacy=0x7f399c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 shutdown_connections 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.918+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 >> 172.21.15.26:0/1595933086 conn(0x7f399c0b7ea0 msgr2=0x7f399c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.919+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 shutdown_connections 2024-03-25T14:31:27.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.919+0000 7f39a2124640 1 -- 172.21.15.26:0/1595933086 wait complete. 2024-03-25T14:31:27.925 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:27.919+0000 7f39a2124640 1 librados: shutdown 2024-03-25T14:31:27.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:34.741 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:34.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.852+0000 7f4cc064e640 10 monclient: get_monmap_and_config 2024-03-25T14:31:34.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.852+0000 7f4cc064e640 10 monclient: build_initial_monmap 2024-03-25T14:31:34.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.852+0000 7f4cc064e640 10 monclient: monmap: 2024-03-25T14:31:34.858 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:34.858 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:34.854498+0000 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:34.854498+0000 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:34.859 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:34.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.853+0000 7f4cc064e640 1 Processor -- start 2024-03-25T14:31:34.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.853+0000 7f4cc064e640 1 -- start start 2024-03-25T14:31:34.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.853+0000 7f4cc064e640 10 monclient: init 2024-03-25T14:31:34.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb80a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb81430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb8142520 con 0x7f4cb813f710 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80b28a0 con 0x7f4cb80a98f0 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80b2a30 con 0x7f4cb81430c0 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:34.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.854+0000 7f4cc064e640 10 monclient(hunting): authenticate will time out at 1779.957880s 2024-03-25T14:31:34.866 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.860+0000 7f4cbe3ea640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4cb80a98f0 0x7f4cb80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:38360/0 (socket says 172.21.15.26:38360) 2024-03-25T14:31:34.866 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.860+0000 7f4cbe3ea640 1 -- 172.21.15.26:0/503965564 learned_addr learned my addr 172.21.15.26:0/503965564 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:34.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.867+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284793058 0 0) 0x7f4cb80b28a0 con 0x7f4cb80a98f0 2024-03-25T14:31:34.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.867+0000 7f4cbd3e8640 10 monclient(hunting): _negotiate 2024-03-25T14:31:34.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.867+0000 7f4cbd3e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:34.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.867+0000 7f4cbd3e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:34.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.867+0000 7f4cbd3e8640 10 monclient(hunting): my global_id is 4417 2024-03-25T14:31:34.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.867+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca4003520 con 0x7f4cb80a98f0 2024-03-25T14:31:34.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3134663813 0 0) 0x7f4ca4003520 con 0x7f4cb80a98f0 2024-03-25T14:31:34.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 >> v1:172.21.15.26:6790/0 conn(0x7f4cb81430c0 legacy=0x7f4cb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:31:34.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 >> v1:172.21.15.26:6789/0 conn(0x7f4cb813f710 legacy=0x7f4cb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:34.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: _finish_hunting 0 2024-03-25T14:31:34.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 1 monclient: found mon.noname-c 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb80b44c0 con 0x7f4cb80a98f0 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: _finish_auth 0 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: _check_auth_tickets 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4cb4001690 con 0x7f4cb80a98f0 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cbd3e8640 10 monclient: dump: 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:34.884 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cc064e640 5 monclient: authenticate success, global_id 4417 2024-03-25T14:31:34.885 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.877+0000 7f4cc064e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cb4001860 con 0x7f4cb80a98f0 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cbd3e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cbd3e8640 1 -- 172.21.15.26:0/503965564 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4cb4001cb0 con 0x7f4cb80a98f0 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cc064e640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cc064e640 4 set_mon_vals no callback set 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cbd3e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cbd3e8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.900+0000 7f4cbd3e8640 10 monclient: dump: 2024-03-25T14:31:34.909 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:34.910 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 10 monclient: shutdown 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 1 -- 172.21.15.26:0/503965564 >> v1:172.21.15.121:6789/0 conn(0x7f4cb80a98f0 legacy=0x7f4cb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 1 -- 172.21.15.26:0/503965564 shutdown_connections 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 1 -- 172.21.15.26:0/503965564 >> 172.21.15.26:0/503965564 conn(0x7f4cb80b7ea0 msgr2=0x7f4cb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 1 -- 172.21.15.26:0/503965564 shutdown_connections 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.901+0000 7f4cc064e640 1 -- 172.21.15.26:0/503965564 wait complete. 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 10 monclient: build_initial_monmap 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 10 monclient: monmap: 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:34.911 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 1 librados: starting msgr at 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 1 librados: starting objecter 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 1 Processor -- start 2024-03-25T14:31:34.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 1 -- start start 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 1 librados: setting wanted keys 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 1 librados: calling monclient init 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.902+0000 7f4cc064e640 10 monclient: init 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb80a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb81430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb81db410 con 0x7f4cb80a98f0 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80ab950 con 0x7f4cb81430c0 2024-03-25T14:31:34.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb80b0df0 con 0x7f4cb813f710 2024-03-25T14:31:34.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:34.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.903+0000 7f4cc064e640 10 monclient(hunting): authenticate will time out at 1780.006844s 2024-03-25T14:31:34.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.908+0000 7f4cbdbe9640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4cb813f710 0x7f4cb80b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45730/0 (socket says 172.21.15.26:45730) 2024-03-25T14:31:34.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.908+0000 7f4cbdbe9640 1 -- 172.21.15.26:0/3400314943 learned_addr learned my addr 172.21.15.26:0/3400314943 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:34.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.909+0000 7f4caeffd640 10 client.?.objecter ms_handle_connect 0x7f4cb813f710 2024-03-25T14:31:34.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.909+0000 7f4caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:34.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.918+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895303408 0 0) 0x7f4cb80b0df0 con 0x7f4cb813f710 2024-03-25T14:31:34.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.918+0000 7f4caeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:31:34.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.918+0000 7f4caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:34.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.918+0000 7f4caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:34.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.918+0000 7f4caeffd640 10 monclient(hunting): my global_id is 4388 2024-03-25T14:31:34.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.918+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c9c004eb0 con 0x7f4cb813f710 2024-03-25T14:31:34.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.921+0000 7f4caeffd640 10 client.?.objecter ms_handle_connect 0x7f4cb81430c0 2024-03-25T14:31:34.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.921+0000 7f4caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:34.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3947914977 0 0) 0x7f4c9c004eb0 con 0x7f4cb813f710 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cb80b0df0 con 0x7f4cb813f710 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ca8001a30 con 0x7f4cb813f710 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 10 monclient(hunting): dump: 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:34.933 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.927+0000 7f4caeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:34.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.929+0000 7f4caeffd640 10 client.?.objecter ms_handle_connect 0x7f4cb80a98f0 2024-03-25T14:31:34.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.929+0000 7f4caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:34.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548587783 0 0) 0x7f4cb80b0df0 con 0x7f4cb813f710 2024-03-25T14:31:34.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 >> v1:172.21.15.121:6789/0 conn(0x7f4cb81430c0 legacy=0x7f4cb81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:34.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 >> v1:172.21.15.26:6789/0 conn(0x7f4cb80a98f0 legacy=0x7f4cb81dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:34.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 10 monclient: _finish_hunting 0 2024-03-25T14:31:34.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 1 monclient: found mon.c 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb80b0f80 con 0x7f4cb813f710 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 10 monclient: _finish_auth 0 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 10 monclient: _check_auth_tickets 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4caeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 5 monclient: authenticate success, global_id 4388 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 10 monclient: _renew_subs 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb80b07f0 con 0x7f4cb813f710 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 10 monclient: _renew_subs 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:34.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb81d48f0 con 0x7f4cb813f710 2024-03-25T14:31:34.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.941+0000 7f4cc064e640 1 librados: init done 2024-03-25T14:31:34.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.942+0000 7f4cc064e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:34.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.942+0000 7f4cc064e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:34.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.942+0000 7f4cc064e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:34.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:34.942+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb80b1190 con 0x7f4cb813f710 2024-03-25T14:31:35.275 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ca8001cf0 con 0x7f4cb813f710 2024-03-25T14:31:35.275 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4caeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:35.275 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4ca8003a60 con 0x7f4cb813f710 2024-03-25T14:31:35.275 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4caeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:35.275 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4caeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4caeffd640 10 monclient: dump: 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:35.276 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.269+0000 7f4cbcbe7640 4 set_mon_vals no callback set 2024-03-25T14:31:35.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.270+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4ca8003c80 con 0x7f4cb813f710 2024-03-25T14:31:35.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.270+0000 7f4cbe3ea640 10 monclient: get_auth_request con 0x7f4c9c06dc90 auth_method 0 2024-03-25T14:31:35.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.271+0000 7f4caeffd640 10 client.4388.objecter ms_handle_connect 0x7f4c9c06dc90 2024-03-25T14:31:35.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.277+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f4ca8054950 con 0x7f4cb813f710 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.277+0000 7f4caeffd640 10 client.4388.objecter ms_dispatch 0x7f4cb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.277+0000 7f4caeffd640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.277+0000 7f4caeffd640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.278+0000 7f4caeffd640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.278+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ca8050290 con 0x7f4cb813f710 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.278+0000 7f4caeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:35.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.278+0000 7f4caeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:35.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.532+0000 7f4cc064e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:35.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.532+0000 7f4cc064e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:35.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.532+0000 7f4cc064e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:35.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.532+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cb80627b0 con 0x7f4cb813f710 2024-03-25T14:31:35.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.557+0000 7f4caeffd640 1 -- 172.21.15.26:0/3400314943 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f4ca8052390 con 0x7f4cb813f710 2024-03-25T14:31:35.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.557+0000 7f4caeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:35.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.557+0000 7f4caeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:35.564 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:35.564 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 10 librados: watch_flush enter 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 10 librados: watch_flush exit 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 10 client.4388.objecter successfully canceled tick 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 >> v1:172.21.15.121:6800/288813066 conn(0x7f4c9c06dc90 legacy=0x7f4c9c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 10 monclient: shutdown 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.559+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 >> v1:172.21.15.26:6790/0 conn(0x7f4cb813f710 legacy=0x7f4cb80b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.560+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 shutdown_connections 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.560+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 >> 172.21.15.26:0/3400314943 conn(0x7f4cb80b7ea0 msgr2=0x7f4cb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:35.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.560+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 shutdown_connections 2024-03-25T14:31:35.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.560+0000 7f4cc064e640 1 -- 172.21.15.26:0/3400314943 wait complete. 2024-03-25T14:31:35.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:35.560+0000 7f4cc064e640 1 librados: shutdown 2024-03-25T14:31:35.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:36.233 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:04:55.094 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.241 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:04:55.102 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.271 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:04:55.131 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.284 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:04:55.145 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.293 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:04:55.170 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.307 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:04:55.168 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.308 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:04:55.184 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.339 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:04:55.215 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:36.343 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:04:55.204 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:42.384 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:42.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.496+0000 7f804ed4b640 10 monclient: get_monmap_and_config 2024-03-25T14:31:42.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.496+0000 7f804ed4b640 10 monclient: build_initial_monmap 2024-03-25T14:31:42.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.496+0000 7f804ed4b640 10 monclient: monmap: 2024-03-25T14:31:42.502 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:42.502 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:42.498347+0000 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:42.498347+0000 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:42.503 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.497+0000 7f804ed4b640 1 Processor -- start 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.497+0000 7f804ed4b640 1 -- start start 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.497+0000 7f804ed4b640 10 monclient: init 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient(hunting): picked mon.noname-b con 0x7f80480b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient(hunting): picked mon.noname-a con 0x7f80480b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient(hunting): picked mon.noname-c con 0x7f804814be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480af940 con 0x7f80480b1730 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480b0a70 con 0x7f804814be90 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480b0d50 con 0x7f80480b4d60 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:42.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.498+0000 7f804ed4b640 10 monclient(hunting): authenticate will time out at 1787.601897s 2024-03-25T14:31:42.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.512+0000 7f804d2e8640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f804814be90 0x7f804814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:59992/0 (socket says 172.21.15.26:59992) 2024-03-25T14:31:42.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.512+0000 7f804d2e8640 1 -- 172.21.15.26:0/160274860 learned_addr learned my addr 172.21.15.26:0/160274860 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:42.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.531+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907332017 0 0) 0x7f80480b0d50 con 0x7f80480b4d60 2024-03-25T14:31:42.537 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.531+0000 7f803f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:42.537 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.531+0000 7f803f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:42.537 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.531+0000 7f803f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:42.537 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.531+0000 7f803f7fe640 10 monclient(hunting): my global_id is 4391 2024-03-25T14:31:42.537 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.531+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8030003520 con 0x7f80480b4d60 2024-03-25T14:31:42.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.537+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974648304 0 0) 0x7f80480b0a70 con 0x7f804814be90 2024-03-25T14:31:42.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.537+0000 7f803f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:42.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.537+0000 7f803f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:42.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.537+0000 7f803f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:42.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.537+0000 7f803f7fe640 10 monclient(hunting): my global_id is 4423 2024-03-25T14:31:42.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.537+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80480b0d50 con 0x7f804814be90 2024-03-25T14:31:42.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.544+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226505828 0 0) 0x7f80480af940 con 0x7f80480b1730 2024-03-25T14:31:42.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.544+0000 7f803f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:42.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.544+0000 7f803f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:42.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.544+0000 7f803f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:42.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.544+0000 7f803f7fe640 10 monclient(hunting): my global_id is 4434 2024-03-25T14:31:42.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.544+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80480b0a70 con 0x7f80480b1730 2024-03-25T14:31:42.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437266146 0 0) 0x7f80480b0a70 con 0x7f80480b1730 2024-03-25T14:31:42.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 >> v1:172.21.15.26:6790/0 conn(0x7f80480b4d60 legacy=0x7f804813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:42.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 >> v1:172.21.15.121:6789/0 conn(0x7f804814be90 legacy=0x7f804814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:42.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 1 monclient: found mon.noname-a 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80480b1030 con 0x7f80480b1730 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f80380015b0 con 0x7f80480b1730 2024-03-25T14:31:42.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f803f7fe640 10 monclient: dump: 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:42.564 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:42.565 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f804ed4b640 5 monclient: authenticate success, global_id 4434 2024-03-25T14:31:42.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.555+0000 7f804ed4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8038001710 con 0x7f80480b1730 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f803f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f803f7fe640 1 -- 172.21.15.26:0/160274860 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8038001b60 con 0x7f80480b1730 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f803f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f803f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f803f7fe640 10 monclient: dump: 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.589 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f804ed4b640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:42.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.583+0000 7f804ed4b640 4 set_mon_vals no callback set 2024-03-25T14:31:42.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 10 monclient: shutdown 2024-03-25T14:31:42.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 1 -- 172.21.15.26:0/160274860 >> v1:172.21.15.26:6789/0 conn(0x7f80480b1730 legacy=0x7f80480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 1 -- 172.21.15.26:0/160274860 shutdown_connections 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 1 -- 172.21.15.26:0/160274860 >> 172.21.15.26:0/160274860 conn(0x7f80480617a0 msgr2=0x7f8048061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 1 -- 172.21.15.26:0/160274860 shutdown_connections 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.584+0000 7f804ed4b640 1 -- 172.21.15.26:0/160274860 wait complete. 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 10 monclient: build_initial_monmap 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 10 monclient: monmap: 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:42.593 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 1 librados: starting msgr at 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 1 librados: starting objecter 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 1 Processor -- start 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 1 -- start start 2024-03-25T14:31:42.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 1 librados: setting wanted keys 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 1 librados: calling monclient init 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.585+0000 7f804ed4b640 10 monclient: init 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient(hunting): picked mon.noname-b con 0x7f80480b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient(hunting): picked mon.noname-a con 0x7f80480b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient(hunting): picked mon.noname-c con 0x7f804814be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80481324b0 con 0x7f80480b4d60 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8048132640 con 0x7f80480b1730 2024-03-25T14:31:42.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80481327d0 con 0x7f804814be90 2024-03-25T14:31:42.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:42.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.586+0000 7f804ed4b640 10 monclient(hunting): authenticate will time out at 1787.690139s 2024-03-25T14:31:42.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.589+0000 7f804cae7640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f80480b1730 0x7f8048144890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60040/0 (socket says 172.21.15.26:60040) 2024-03-25T14:31:42.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.589+0000 7f804cae7640 1 -- 172.21.15.26:0/1447426797 learned_addr learned my addr 172.21.15.26:0/1447426797 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:42.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.591+0000 7f803d7fa640 10 client.?.objecter ms_handle_connect 0x7f80480b1730 2024-03-25T14:31:42.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.591+0000 7f803d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:42.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.600+0000 7f803d7fa640 10 client.?.objecter ms_handle_connect 0x7f804814be90 2024-03-25T14:31:42.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.600+0000 7f803d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:42.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.601+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015965777 0 0) 0x7f8048132640 con 0x7f80480b1730 2024-03-25T14:31:42.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.601+0000 7f803d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:42.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.601+0000 7f803d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:42.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.601+0000 7f803d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:42.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.601+0000 7f803d7fa640 10 monclient(hunting): my global_id is 4426 2024-03-25T14:31:42.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.601+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8034004eb0 con 0x7f80480b1730 2024-03-25T14:31:42.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.603+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928080532 0 0) 0x7f80481327d0 con 0x7f804814be90 2024-03-25T14:31:42.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.603+0000 7f803d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:42.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.603+0000 7f803d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:42.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.603+0000 7f803d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:42.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.603+0000 7f803d7fa640 10 monclient(hunting): my global_id is 4394 2024-03-25T14:31:42.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.603+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8048132640 con 0x7f804814be90 2024-03-25T14:31:42.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.609+0000 7f803d7fa640 10 client.?.objecter ms_handle_connect 0x7f80480b4d60 2024-03-25T14:31:42.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.609+0000 7f803d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:42.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725583068 0 0) 0x7f8034004eb0 con 0x7f80480b1730 2024-03-25T14:31:42.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80481327d0 con 0x7f80480b1730 2024-03-25T14:31:42.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8044001990 con 0x7f80480b1730 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.612+0000 7f803d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.620+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197162772 0 0) 0x7f8048132640 con 0x7f804814be90 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.621+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8034004eb0 con 0x7f804814be90 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.621+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8040001bd0 con 0x7f804814be90 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.621+0000 7f803d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.621+0000 7f803d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.621+0000 7f803d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.627 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.628 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:42.628 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:42.628 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:42.628 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:42.628 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:42.628 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.623+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415286263 0 0) 0x7f80481324b0 con 0x7f80480b4d60 2024-03-25T14:31:42.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.623+0000 7f803d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:42.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.623+0000 7f803d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:42.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.623+0000 7f803d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:42.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.623+0000 7f803d7fa640 10 monclient(hunting): my global_id is 4437 2024-03-25T14:31:42.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.623+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8048132640 con 0x7f80480b4d60 2024-03-25T14:31:42.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2666093726 0 0) 0x7f80481327d0 con 0x7f80480b1730 2024-03-25T14:31:42.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 >> v1:172.21.15.26:6790/0 conn(0x7f804814be90 legacy=0x7f80481d7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:42.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 >> v1:172.21.15.26:6789/0 conn(0x7f80480b4d60 legacy=0x7f80481d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:42.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:31:42.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 1 monclient: found mon.b 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048138e60 con 0x7f80480b1730 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f803d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 5 monclient: authenticate success, global_id 4426 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 10 monclient: _renew_subs 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8048139070 con 0x7f80480b1730 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 10 monclient: _renew_subs 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:42.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8048137a80 con 0x7f80480b1730 2024-03-25T14:31:42.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.625+0000 7f804ed4b640 1 librados: init done 2024-03-25T14:31:42.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.626+0000 7f804ed4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:42.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.626+0000 7f804ed4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:42.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.626+0000 7f804ed4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:42.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.626+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80481d7730 con 0x7f80480b1730 2024-03-25T14:31:42.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8044003610 con 0x7f80480b1730 2024-03-25T14:31:42.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:42.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f80440039a0 con 0x7f80480b1730 2024-03-25T14:31:42.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:42.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803effd640 4 set_mon_vals no callback set 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.892+0000 7f803d7fa640 10 monclient: dump: 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:42.899 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:42.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.893+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f8044003bc0 con 0x7f80480b1730 2024-03-25T14:31:42.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.893+0000 7f803ffff640 10 monclient: get_auth_request con 0x7f803406df50 auth_method 0 2024-03-25T14:31:42.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.894+0000 7f803d7fa640 10 client.4426.objecter ms_handle_connect 0x7f803406df50 2024-03-25T14:31:42.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.898+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f80440547f0 con 0x7f80480b1730 2024-03-25T14:31:42.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.898+0000 7f803d7fa640 10 client.4426.objecter ms_dispatch 0x7f8048000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:42.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.898+0000 7f803d7fa640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:42.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.898+0000 7f803d7fa640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:42.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.898+0000 7f803d7fa640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T14:31:42.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.899+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8044050300 con 0x7f80480b1730 2024-03-25T14:31:42.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.899+0000 7f803d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:42.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:42.899+0000 7f803d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:43.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.125+0000 7f804ed4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:43.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.125+0000 7f804ed4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:43.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.125+0000 7f804ed4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:31:43.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.125+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f804813a950 con 0x7f80480b1730 2024-03-25T14:31:43.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.157+0000 7f803d7fa640 1 -- 172.21.15.26:0/1447426797 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f804813a950 con 0x7f80480b1730 2024-03-25T14:31:43.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.157+0000 7f803d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:43.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.157+0000 7f803d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:43.208 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:43.208 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 10 librados: watch_flush enter 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 10 librados: watch_flush exit 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 10 client.4426.objecter successfully canceled tick 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 >> v1:172.21.15.121:6800/288813066 conn(0x7f803406df50 legacy=0x7f8034070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 10 monclient: shutdown 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 >> v1:172.21.15.121:6789/0 conn(0x7f80480b1730 legacy=0x7f8048144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:43.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 shutdown_connections 2024-03-25T14:31:43.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 >> 172.21.15.26:0/1447426797 conn(0x7f80480617a0 msgr2=0x7f80480b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:43.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 shutdown_connections 2024-03-25T14:31:43.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 -- 172.21.15.26:0/1447426797 wait complete. 2024-03-25T14:31:43.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:43.160+0000 7f804ed4b640 1 librados: shutdown 2024-03-25T14:31:43.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:50.018 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.130+0000 7f49adc8d640 10 monclient: get_monmap_and_config 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.130+0000 7f49adc8d640 10 monclient: build_initial_monmap 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.130+0000 7f49adc8d640 10 monclient: monmap: 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:50.131066+0000 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:50.131066+0000 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:50.135 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:50.136 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:50.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.131+0000 7f49adc8d640 1 Processor -- start 2024-03-25T14:31:50.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.131+0000 7f49adc8d640 1 -- start start 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.131+0000 7f49adc8d640 10 monclient: init 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient(hunting): picked mon.noname-c con 0x7f49a80a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient(hunting): picked mon.noname-a con 0x7f49a813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient(hunting): picked mon.noname-b con 0x7f49a81430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a8142520 con 0x7f49a813f710 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a80b28a0 con 0x7f49a80a98f0 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a80b2a30 con 0x7f49a81430c0 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:50.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.132+0000 7f49adc8d640 10 monclient(hunting): authenticate will time out at 1795.234793s 2024-03-25T14:31:50.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.151+0000 7f49a6ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f49a813f710 0x7f49a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43438/0 (socket says 172.21.15.26:43438) 2024-03-25T14:31:50.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.151+0000 7f49a6ffd640 1 -- 172.21.15.26:0/2080328129 learned_addr learned my addr 172.21.15.26:0/2080328129 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:50.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.174+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916031557 0 0) 0x7f49a8142520 con 0x7f49a813f710 2024-03-25T14:31:50.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.174+0000 7f49a67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:31:50.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.174+0000 7f49a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:50.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.174+0000 7f49a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:50.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.174+0000 7f49a67fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T14:31:50.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.174+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4990003520 con 0x7f49a813f710 2024-03-25T14:31:50.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.178+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674474021 0 0) 0x7f49a80b28a0 con 0x7f49a80a98f0 2024-03-25T14:31:50.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.178+0000 7f49a67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:31:50.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.178+0000 7f49a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:50.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.178+0000 7f49a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:50.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.178+0000 7f49a67fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T14:31:50.183 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.178+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a8142520 con 0x7f49a80a98f0 2024-03-25T14:31:50.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.180+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174424230 0 0) 0x7f49a80b2a30 con 0x7f49a81430c0 2024-03-25T14:31:50.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.180+0000 7f49a67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:31:50.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.180+0000 7f49a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:50.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.180+0000 7f49a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:50.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.180+0000 7f49a67fc640 10 monclient(hunting): my global_id is 4397 2024-03-25T14:31:50.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.180+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a80b28a0 con 0x7f49a81430c0 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1538416036 0 0) 0x7f4990003520 con 0x7f49a813f710 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 >> v1:172.21.15.26:6790/0 conn(0x7f49a81430c0 legacy=0x7f49a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 >> v1:172.21.15.121:6789/0 conn(0x7f49a80a98f0 legacy=0x7f49a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 1 monclient: found mon.noname-a 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:31:50.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a80b44c0 con 0x7f49a813f710 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: _finish_auth 0 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f499c001690 con 0x7f49a813f710 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49adc8d640 5 monclient: authenticate success, global_id 4440 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49adc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49a67fc640 10 monclient: dump: 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.192 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:50.193 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:50.193 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:50.193 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:50.193 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:50.193 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:50.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.186+0000 7f49adc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:50.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f499c0017f0 con 0x7f49a813f710 2024-03-25T14:31:50.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:50.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49a67fc640 1 -- 172.21.15.26:0/2080328129 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f499c001c00 con 0x7f49a813f710 2024-03-25T14:31:50.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49adc8d640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:50.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49adc8d640 4 set_mon_vals no callback set 2024-03-25T14:31:50.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.200+0000 7f49a67fc640 10 monclient: dump: 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:50.206 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:50.207 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 10 monclient: shutdown 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 1 -- 172.21.15.26:0/2080328129 >> v1:172.21.15.26:6789/0 conn(0x7f49a813f710 legacy=0x7f49a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 1 -- 172.21.15.26:0/2080328129 shutdown_connections 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 1 -- 172.21.15.26:0/2080328129 >> 172.21.15.26:0/2080328129 conn(0x7f49a80b7ea0 msgr2=0x7f49a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 1 -- 172.21.15.26:0/2080328129 shutdown_connections 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.201+0000 7f49adc8d640 1 -- 172.21.15.26:0/2080328129 wait complete. 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.202+0000 7f49adc8d640 10 monclient: build_initial_monmap 2024-03-25T14:31:50.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.202+0000 7f49adc8d640 10 monclient: monmap: 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.202+0000 7f49adc8d640 1 librados: starting msgr at 2024-03-25T14:31:50.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.202+0000 7f49adc8d640 1 librados: starting objecter 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 Processor -- start 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 -- start start 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 librados: setting wanted keys 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 librados: calling monclient init 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient: init 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient(hunting): picked mon.noname-c con 0x7f49a80a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient(hunting): picked mon.noname-b con 0x7f49a813f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient(hunting): picked mon.noname-a con 0x7f49a81430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a8137f10 con 0x7f49a81430c0 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a81dc070 con 0x7f49a813f710 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a8137100 con 0x7f49a80a98f0 2024-03-25T14:31:50.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:50.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.203+0000 7f49adc8d640 10 monclient(hunting): authenticate will time out at 1795.306516s 2024-03-25T14:31:50.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.217+0000 7f49a6ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f49a813f710 0x7f49a81db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:35462/0 (socket says 172.21.15.26:35462) 2024-03-25T14:31:50.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.217+0000 7f49a6ffd640 1 -- 172.21.15.26:0/725242138 learned_addr learned my addr 172.21.15.26:0/725242138 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:50.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.217+0000 7f498ffff640 10 client.?.objecter ms_handle_connect 0x7f49a813f710 2024-03-25T14:31:50.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.217+0000 7f498ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:50.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.220+0000 7f498ffff640 10 client.?.objecter ms_handle_connect 0x7f49a80a98f0 2024-03-25T14:31:50.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.220+0000 7f498ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:50.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.224+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796139052 0 0) 0x7f49a8137100 con 0x7f49a80a98f0 2024-03-25T14:31:50.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.224+0000 7f498ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:31:50.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.224+0000 7f498ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:50.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.224+0000 7f498ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:50.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.224+0000 7f498ffff640 10 monclient(hunting): my global_id is 4400 2024-03-25T14:31:50.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.224+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4994004eb0 con 0x7f49a80a98f0 2024-03-25T14:31:50.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.230+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710244285 0 0) 0x7f49a81dc070 con 0x7f49a813f710 2024-03-25T14:31:50.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.230+0000 7f498ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:31:50.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.230+0000 7f498ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:50.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.230+0000 7f498ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:50.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.230+0000 7f498ffff640 10 monclient(hunting): my global_id is 4432 2024-03-25T14:31:50.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.230+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a8137100 con 0x7f49a813f710 2024-03-25T14:31:50.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.234+0000 7f498ffff640 10 client.?.objecter ms_handle_connect 0x7f49a81430c0 2024-03-25T14:31:50.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.234+0000 7f498ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160949917 0 0) 0x7f4994004eb0 con 0x7f49a80a98f0 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49a81dc070 con 0x7f49a80a98f0 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4998001970 con 0x7f49a80a98f0 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 10 monclient(hunting): dump: 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:50.243 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:50.244 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.238+0000 7f498ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:50.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273396692 0 0) 0x7f49a81dc070 con 0x7f49a80a98f0 2024-03-25T14:31:50.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 >> v1:172.21.15.121:6789/0 conn(0x7f49a813f710 legacy=0x7f49a81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 >> v1:172.21.15.26:6789/0 conn(0x7f49a81430c0 legacy=0x7f49a81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 1 monclient: found mon.c 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a81373e0 con 0x7f49a80a98f0 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 10 monclient: _finish_auth 0 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f498ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 5 monclient: authenticate success, global_id 4400 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 10 monclient: _renew_subs 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a81358e0 con 0x7f49a80a98f0 2024-03-25T14:31:50.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:50.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 10 monclient: _renew_subs 2024-03-25T14:31:50.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:50.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a8136150 con 0x7f49a80a98f0 2024-03-25T14:31:50.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.246+0000 7f49adc8d640 1 librados: init done 2024-03-25T14:31:50.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.248+0000 7f49adc8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:50.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.248+0000 7f49adc8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:50.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.248+0000 7f49adc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:50.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.248+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a80627b0 con 0x7f49a80a98f0 2024-03-25T14:31:50.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.511+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4998003610 con 0x7f49a80a98f0 2024-03-25T14:31:50.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.511+0000 7f498ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:50.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.511+0000 7f49a5ffb640 4 set_mon_vals no callback set 2024-03-25T14:31:50.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.539+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4998003920 con 0x7f49a80a98f0 2024-03-25T14:31:50.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.539+0000 7f498ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:50.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.539+0000 7f498ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:50.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.539+0000 7f498ffff640 10 monclient: dump: 2024-03-25T14:31:50.544 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:50.544 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:50.545 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:50.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.540+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4998019970 con 0x7f49a80a98f0 2024-03-25T14:31:50.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.541+0000 7f49a6ffd640 10 monclient: get_auth_request con 0x7f499406dd10 auth_method 0 2024-03-25T14:31:50.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.541+0000 7f498ffff640 10 client.4400.objecter ms_handle_connect 0x7f499406dd10 2024-03-25T14:31:50.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f49980536b0 con 0x7f49a80a98f0 2024-03-25T14:31:50.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 10 client.4400.objecter ms_dispatch 0x7f49a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:50.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:50.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:50.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T14:31:50.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f499804eff0 con 0x7f49a80a98f0 2024-03-25T14:31:50.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:50.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.546+0000 7f498ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:50.773 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.768+0000 7f49adc8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:50.773 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.768+0000 7f49adc8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:50.773 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.768+0000 7f49adc8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:50.774 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.768+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49a813e250 con 0x7f49a80a98f0 2024-03-25T14:31:50.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.838+0000 7f498ffff640 1 -- 172.21.15.26:0/725242138 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f49980510f0 con 0x7f49a80a98f0 2024-03-25T14:31:50.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.838+0000 7f498ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:50.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.838+0000 7f498ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:50.844 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:50.844 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 10 librados: watch_flush enter 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 10 librados: watch_flush exit 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 10 client.4400.objecter successfully canceled tick 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 >> v1:172.21.15.121:6800/288813066 conn(0x7f499406dd10 legacy=0x7f49940701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 10 monclient: shutdown 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.840+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 >> v1:172.21.15.26:6790/0 conn(0x7f49a80a98f0 legacy=0x7f49a8137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.841+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 shutdown_connections 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.841+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 >> 172.21.15.26:0/725242138 conn(0x7f49a80b7ea0 msgr2=0x7f49a814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:50.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.841+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 shutdown_connections 2024-03-25T14:31:50.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.841+0000 7f49adc8d640 1 -- 172.21.15.26:0/725242138 wait complete. 2024-03-25T14:31:50.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:50.841+0000 7f49adc8d640 1 librados: shutdown 2024-03-25T14:31:50.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:56.274 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:15.133 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.330 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:15.191 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.373 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:15.250 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.374 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:15.234 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.381 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:15.242 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.387 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:15.263 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.394 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:15.255 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.425 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:15.301 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.435 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:15.295 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:57.666 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.783+0000 7f52555a0640 10 monclient: get_monmap_and_config 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.783+0000 7f52555a0640 10 monclient: build_initial_monmap 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.783+0000 7f52555a0640 10 monclient: monmap: 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:31:57.784011+0000 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:31:57.784011+0000 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:57.788 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:57.789 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:31:57.789 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:31:57.789 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:57.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.784+0000 7f52555a0640 1 Processor -- start 2024-03-25T14:31:57.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.784+0000 7f52555a0640 1 -- start start 2024-03-25T14:31:57.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.784+0000 7f52555a0640 10 monclient: init 2024-03-25T14:31:57.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:57.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient(hunting): picked mon.noname-c con 0x7f52500b48e0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient(hunting): picked mon.noname-b con 0x7f525013fc60 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient(hunting): picked mon.noname-a con 0x7f52500b3f90 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500a9d10 con 0x7f52500b3f90 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500ab000 con 0x7f52500b48e0 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500a9680 con 0x7f525013fc60 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:57.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.785+0000 7f52555a0640 10 monclient(hunting): authenticate will time out at 1802.888096s 2024-03-25T14:31:57.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.795+0000 7f524ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f52500b48e0 0x7f525013f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:37618/0 (socket says 172.21.15.26:37618) 2024-03-25T14:31:57.800 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.795+0000 7f524ffff640 1 -- 172.21.15.26:0/2453854722 learned_addr learned my addr 172.21.15.26:0/2453854722 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:57.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.804+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835136485 0 0) 0x7f52500ab000 con 0x7f52500b48e0 2024-03-25T14:31:57.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.804+0000 7f524f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:57.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.804+0000 7f524f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:57.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.804+0000 7f524f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:57.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.804+0000 7f524f7fe640 10 monclient(hunting): my global_id is 4435 2024-03-25T14:31:57.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.804+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c003520 con 0x7f52500b48e0 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937295753 0 0) 0x7f523c003520 con 0x7f52500b48e0 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 >> v1:172.21.15.26:6790/0 conn(0x7f525013fc60 legacy=0x7f5250145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 >> v1:172.21.15.26:6789/0 conn(0x7f52500b3f90 legacy=0x7f525014dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 1 monclient: found mon.noname-c 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500a9810 con 0x7f52500b48e0 2024-03-25T14:31:57.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5240001690 con 0x7f52500b48e0 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f524f7fe640 10 monclient: dump: 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:57.821 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:57.822 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:57.822 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:57.822 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f52555a0640 5 monclient: authenticate success, global_id 4435 2024-03-25T14:31:57.822 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.814+0000 7f52555a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:57.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5240001860 con 0x7f52500b48e0 2024-03-25T14:31:57.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f524f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:57.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f524f7fe640 1 -- 172.21.15.26:0/2453854722 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5240001cb0 con 0x7f52500b48e0 2024-03-25T14:31:57.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f52555a0640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:57.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f52555a0640 4 set_mon_vals no callback set 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f524f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f524f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.826+0000 7f524f7fe640 10 monclient: dump: 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:57.832 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 10 monclient: shutdown 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 1 -- 172.21.15.26:0/2453854722 >> v1:172.21.15.121:6789/0 conn(0x7f52500b48e0 legacy=0x7f525013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 1 -- 172.21.15.26:0/2453854722 shutdown_connections 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 1 -- 172.21.15.26:0/2453854722 >> 172.21.15.26:0/2453854722 conn(0x7f52500615c0 msgr2=0x7f52500619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 1 -- 172.21.15.26:0/2453854722 shutdown_connections 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.827+0000 7f52555a0640 1 -- 172.21.15.26:0/2453854722 wait complete. 2024-03-25T14:31:57.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.828+0000 7f52555a0640 10 monclient: build_initial_monmap 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.828+0000 7f52555a0640 10 monclient: monmap: 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 1 librados: starting msgr at 2024-03-25T14:31:57.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 1 librados: starting objecter 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 1 Processor -- start 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 1 -- start start 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 1 librados: setting wanted keys 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 1 librados: calling monclient init 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.829+0000 7f52555a0640 10 monclient: init 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient(hunting): picked mon.noname-a con 0x7f52500b3f90 addr v1:172.21.15.26:6789/0 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient(hunting): picked mon.noname-b con 0x7f52500b48e0 addr v1:172.21.15.121:6789/0 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient(hunting): picked mon.noname-c con 0x7f525013fc60 addr v1:172.21.15.26:6790/0 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500ad380 con 0x7f52500b3f90 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500ad660 con 0x7f52500b48e0 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5250145530 con 0x7f525013fc60 2024-03-25T14:31:57.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:57.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.830+0000 7f52555a0640 10 monclient(hunting): authenticate will time out at 1802.932915s 2024-03-25T14:31:57.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.836+0000 7f524ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f52500b3f90 0x7f52500b10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58184/0 (socket says 172.21.15.26:58184) 2024-03-25T14:31:57.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.836+0000 7f524ffff640 1 -- 172.21.15.26:0/1223631306 learned_addr learned my addr 172.21.15.26:0/1223631306 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:31:57.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.839+0000 7f524d7fa640 10 client.?.objecter ms_handle_connect 0x7f525013fc60 2024-03-25T14:31:57.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.839+0000 7f524d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:57.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.839+0000 7f524d7fa640 10 client.?.objecter ms_handle_connect 0x7f52500b3f90 2024-03-25T14:31:57.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.839+0000 7f524d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:57.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.848+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354817373 0 0) 0x7f5250145530 con 0x7f525013fc60 2024-03-25T14:31:57.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.848+0000 7f524d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:57.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.848+0000 7f524d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:57.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.848+0000 7f524d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:57.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.848+0000 7f524d7fa640 10 monclient(hunting): my global_id is 4403 2024-03-25T14:31:57.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.848+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5230004eb0 con 0x7f525013fc60 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 10 client.?.objecter ms_handle_connect 0x7f52500b48e0 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727126293 0 0) 0x7f52500ad380 con 0x7f52500b3f90 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 10 monclient(hunting): my global_id is 4449 2024-03-25T14:31:57.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.852+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5250145530 con 0x7f52500b3f90 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.863+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369939919 0 0) 0x7f52500ad660 con 0x7f52500b48e0 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.863+0000 7f524d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.863+0000 7f524d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.863+0000 7f524d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.863+0000 7f524d7fa640 10 monclient(hunting): my global_id is 4438 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.863+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52500ad380 con 0x7f52500b48e0 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328497899 0 0) 0x7f5230004eb0 con 0x7f525013fc60 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52500ad660 con 0x7f525013fc60 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5248001c30 con 0x7f525013fc60 2024-03-25T14:31:57.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:57.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.864+0000 7f524d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:57.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3677087808 0 0) 0x7f52500ad660 con 0x7f525013fc60 2024-03-25T14:31:57.876 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 >> v1:172.21.15.121:6789/0 conn(0x7f52500b48e0 legacy=0x7f52501d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 >> v1:172.21.15.26:6789/0 conn(0x7f52500b3f90 legacy=0x7f52500b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 1 monclient: found mon.c 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500ac7b0 con 0x7f525013fc60 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f524d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 5 monclient: authenticate success, global_id 4403 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 10 monclient: _renew_subs 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:57.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5250138a20 con 0x7f525013fc60 2024-03-25T14:31:57.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:57.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 10 monclient: _renew_subs 2024-03-25T14:31:57.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:57.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52500abb70 con 0x7f525013fc60 2024-03-25T14:31:57.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.871+0000 7f52555a0640 1 librados: init done 2024-03-25T14:31:57.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.872+0000 7f52555a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:57.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.872+0000 7f52555a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:57.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.872+0000 7f52555a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:57.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.872+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52500aa180 con 0x7f525013fc60 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5248003990 con 0x7f525013fc60 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5248003c00 con 0x7f525013fc60 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524d7fa640 10 monclient: dump: 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.921 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:31:57.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.916+0000 7f524effd640 4 set_mon_vals no callback set 2024-03-25T14:31:57.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.917+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f5248003e20 con 0x7f525013fc60 2024-03-25T14:31:57.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.917+0000 7f524ffff640 10 monclient: get_auth_request con 0x7f523006de20 auth_method 0 2024-03-25T14:31:57.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:57.918+0000 7f524d7fa640 10 client.4403.objecter ms_handle_connect 0x7f523006de20 2024-03-25T14:31:58.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f5248053730 con 0x7f525013fc60 2024-03-25T14:31:58.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 10 client.4403.objecter ms_dispatch 0x7f5250000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:31:58.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:31:58.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:31:58.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-25T14:31:58.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f524804f240 con 0x7f525013fc60 2024-03-25T14:31:58.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:58.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.170+0000 7f524d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:58.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.424+0000 7f52555a0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:58.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.424+0000 7f52555a0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:58.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.424+0000 7f52555a0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:31:58.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.424+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5250138bb0 con 0x7f525013fc60 2024-03-25T14:31:58.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.453+0000 7f524d7fa640 1 -- 172.21.15.26:0/1223631306 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f5250138bb0 con 0x7f525013fc60 2024-03-25T14:31:58.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.453+0000 7f524d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:58.458 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.453+0000 7f524d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:58.458 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:31:58.459 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 10 librados: watch_flush enter 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 10 librados: watch_flush exit 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 10 client.4403.objecter successfully canceled tick 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 >> v1:172.21.15.121:6800/288813066 conn(0x7f523006de20 legacy=0x7f52300702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 10 monclient: shutdown 2024-03-25T14:31:58.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:58.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 >> v1:172.21.15.26:6790/0 conn(0x7f525013fc60 legacy=0x7f52501e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:58.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 shutdown_connections 2024-03-25T14:31:58.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 >> 172.21.15.26:0/1223631306 conn(0x7f52500615c0 msgr2=0x7f52500b5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:58.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 shutdown_connections 2024-03-25T14:31:58.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 -- 172.21.15.26:0/1223631306 wait complete. 2024-03-25T14:31:58.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:31:58.455+0000 7f52555a0640 1 librados: shutdown 2024-03-25T14:31:58.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:05.279 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:05.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.392+0000 7fe0f14c9640 10 monclient: get_monmap_and_config 2024-03-25T14:32:05.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.392+0000 7fe0f14c9640 10 monclient: build_initial_monmap 2024-03-25T14:32:05.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.392+0000 7fe0f14c9640 10 monclient: monmap: 2024-03-25T14:32:05.397 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:05.398 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.398 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:05.393501+0000 2024-03-25T14:32:05.398 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:05.393501+0000 2024-03-25T14:32:05.398 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:05.398 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.398 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:05.399 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:05.399 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:05.399 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.393+0000 7fe0f14c9640 1 Processor -- start 2024-03-25T14:32:05.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.393+0000 7fe0f14c9640 1 -- start start 2024-03-25T14:32:05.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.393+0000 7fe0f14c9640 10 monclient: init 2024-03-25T14:32:05.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:05.400 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ec06c4a0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ec06d720 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ec0aef30 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec0ade40 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec0ae120 con 0x7fe0ec06d720 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec06d2d0 con 0x7fe0ec0aef30 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:05.401 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.394+0000 7fe0f14c9640 10 monclient(hunting): authenticate will time out at 1810.497137s 2024-03-25T14:32:05.406 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.401+0000 7fe0ea7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe0ec06d720 0x7fe0ec0ae820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42260/0 (socket says 172.21.15.26:42260) 2024-03-25T14:32:05.406 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.402+0000 7fe0ea7fc640 1 -- 172.21.15.26:0/3564274530 learned_addr learned my addr 172.21.15.26:0/3564274530 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:05.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.408+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019451017 0 0) 0x7fe0ec0ae120 con 0x7fe0ec06d720 2024-03-25T14:32:05.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.408+0000 7fe0e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:32:05.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.408+0000 7fe0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:05.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.408+0000 7fe0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:05.413 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.408+0000 7fe0e9ffb640 10 monclient(hunting): my global_id is 4441 2024-03-25T14:32:05.414 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.408+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4003520 con 0x7fe0ec06d720 2024-03-25T14:32:05.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.418+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167137193 0 0) 0x7fe0ec0ade40 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.418+0000 7fe0e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:32:05.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.418+0000 7fe0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:05.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.418+0000 7fe0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:05.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.418+0000 7fe0e9ffb640 10 monclient(hunting): my global_id is 4452 2024-03-25T14:32:05.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.418+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec0ae120 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.421+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621606291 0 0) 0x7fe0ec06d2d0 con 0x7fe0ec0aef30 2024-03-25T14:32:05.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.421+0000 7fe0e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:32:05.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.421+0000 7fe0e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:05.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.421+0000 7fe0e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:05.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.421+0000 7fe0e9ffb640 10 monclient(hunting): my global_id is 4406 2024-03-25T14:32:05.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.421+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec0ade40 con 0x7fe0ec0aef30 2024-03-25T14:32:05.432 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142153798 0 0) 0x7fe0d4003520 con 0x7fe0ec06d720 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 >> v1:172.21.15.26:6790/0 conn(0x7fe0ec0aef30 legacy=0x7fe0ec14e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 >> v1:172.21.15.26:6789/0 conn(0x7fe0ec06c4a0 legacy=0x7fe0ec0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 1 monclient: found mon.noname-c 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec06cbc0 con 0x7fe0ec06d720 2024-03-25T14:32:05.433 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: _finish_auth 0 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe0e00016e0 con 0x7fe0ec06d720 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0e9ffb640 10 monclient: dump: 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.434 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0f14c9640 5 monclient: authenticate success, global_id 4441 2024-03-25T14:32:05.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.427+0000 7fe0f14c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0e0001840 con 0x7fe0ec06d720 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0e9ffb640 1 -- 172.21.15.26:0/3564274530 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe0e0001bd0 con 0x7fe0ec06d720 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0f14c9640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0f14c9640 4 set_mon_vals no callback set 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.444+0000 7fe0e9ffb640 10 monclient: dump: 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.450 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:05.451 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 10 monclient: shutdown 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 1 -- 172.21.15.26:0/3564274530 >> v1:172.21.15.121:6789/0 conn(0x7fe0ec06d720 legacy=0x7fe0ec0ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 1 -- 172.21.15.26:0/3564274530 shutdown_connections 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 1 -- 172.21.15.26:0/3564274530 >> 172.21.15.26:0/3564274530 conn(0x7fe0ec0b5d10 msgr2=0x7fe0ec0b6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 1 -- 172.21.15.26:0/3564274530 shutdown_connections 2024-03-25T14:32:05.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.445+0000 7fe0f14c9640 1 -- 172.21.15.26:0/3564274530 wait complete. 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.446+0000 7fe0f14c9640 10 monclient: build_initial_monmap 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.446+0000 7fe0f14c9640 10 monclient: monmap: 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:05.454 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.446+0000 7fe0f14c9640 1 librados: starting msgr at 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.446+0000 7fe0f14c9640 1 librados: starting objecter 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 Processor -- start 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 -- start start 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 librados: setting wanted keys 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 librados: calling monclient init 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient: init 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ec06c4a0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ec06d720 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ec0aef30 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec1c2a30 con 0x7fe0ec0aef30 2024-03-25T14:32:05.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec1c42f0 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec1c4560 con 0x7fe0ec06d720 2024-03-25T14:32:05.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:05.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.447+0000 7fe0f14c9640 10 monclient(hunting): authenticate will time out at 1810.550363s 2024-03-25T14:32:05.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.460+0000 7fe0eaffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe0ec06c4a0 0x7fe0ec1dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42316/0 (socket says 172.21.15.26:42316) 2024-03-25T14:32:05.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.460+0000 7fe0eaffd640 1 -- 172.21.15.26:0/1261341811 learned_addr learned my addr 172.21.15.26:0/1261341811 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:05.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.461+0000 7fe0d37fe640 10 client.?.objecter ms_handle_connect 0x7fe0ec06c4a0 2024-03-25T14:32:05.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.461+0000 7fe0d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:05.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.468+0000 7fe0d37fe640 10 client.?.objecter ms_handle_connect 0x7fe0ec0aef30 2024-03-25T14:32:05.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.468+0000 7fe0d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:05.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.482+0000 7fe0d37fe640 10 client.?.objecter ms_handle_connect 0x7fe0ec06d720 2024-03-25T14:32:05.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.482+0000 7fe0d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:05.489 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.484+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952144113 0 0) 0x7fe0ec1c2a30 con 0x7fe0ec0aef30 2024-03-25T14:32:05.489 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.484+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:32:05.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.484+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:05.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.484+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:05.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.484+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4455 2024-03-25T14:32:05.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.484+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d8004eb0 con 0x7fe0ec0aef30 2024-03-25T14:32:05.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714701578 0 0) 0x7fe0ec1c42f0 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:32:05.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:05.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:05.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4444 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec1c2a30 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046374235 0 0) 0x7fe0ec1c4560 con 0x7fe0ec06d720 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4409 2024-03-25T14:32:05.494 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.488+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec1c42f0 con 0x7fe0ec06d720 2024-03-25T14:32:05.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858338420 0 0) 0x7fe0d8004eb0 con 0x7fe0ec0aef30 2024-03-25T14:32:05.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0ec1c4560 con 0x7fe0ec0aef30 2024-03-25T14:32:05.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe0e4001bd0 con 0x7fe0ec0aef30 2024-03-25T14:32:05.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 10 monclient(hunting): dump: 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.503+0000 7fe0d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:05.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.505+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 695989471 0 0) 0x7fe0ec1c42f0 con 0x7fe0ec06d720 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.505+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d8004eb0 con 0x7fe0ec06d720 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.505+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe0e00019b0 con 0x7fe0ec06d720 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.505+0000 7fe0d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.505+0000 7fe0d37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.505+0000 7fe0d37fe640 10 monclient(hunting): dump: 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.514 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.506+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112765798 0 0) 0x7fe0ec1c2a30 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.506+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0ec1c42f0 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.506+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe0dc0019f0 con 0x7fe0ec06c4a0 2024-03-25T14:32:05.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.506+0000 7fe0d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.506+0000 7fe0d37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.506+0000 7fe0d37fe640 10 monclient(hunting): dump: 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:05.516 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:05.517 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3262887821 0 0) 0x7fe0d8004eb0 con 0x7fe0ec06d720 2024-03-25T14:32:05.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 >> v1:172.21.15.121:6789/0 conn(0x7fe0ec06c4a0 legacy=0x7fe0ec1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:05.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 >> v1:172.21.15.26:6789/0 conn(0x7fe0ec0aef30 legacy=0x7fe0ec1c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:05.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 10 monclient: _finish_hunting 0 2024-03-25T14:32:05.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 1 monclient: found mon.c 2024-03-25T14:32:05.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec1c2610 con 0x7fe0ec06d720 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 10 monclient: _finish_auth 0 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 10 monclient: _check_auth_tickets 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 5 monclient: authenticate success, global_id 4409 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 10 monclient: _renew_subs 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec1c4980 con 0x7fe0ec06d720 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 10 monclient: _renew_subs 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:05.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec1c07e0 con 0x7fe0ec06d720 2024-03-25T14:32:05.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.513+0000 7fe0f14c9640 1 librados: init done 2024-03-25T14:32:05.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.514+0000 7fe0f14c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:05.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.514+0000 7fe0f14c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:05.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.514+0000 7fe0f14c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:05.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.514+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec1c2860 con 0x7fe0ec06d720 2024-03-25T14:32:05.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.727+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0e0001bd0 con 0x7fe0ec06d720 2024-03-25T14:32:05.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.727+0000 7fe0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:05.732 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.727+0000 7fe0e97fa640 4 set_mon_vals no callback set 2024-03-25T14:32:05.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.831+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe0e0004c30 con 0x7fe0ec06d720 2024-03-25T14:32:05.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.831+0000 7fe0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:05.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.831+0000 7fe0d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:05.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.831+0000 7fe0d37fe640 10 monclient: dump: 2024-03-25T14:32:05.836 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:05.837 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:05.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.832+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fe0e001ac00 con 0x7fe0ec06d720 2024-03-25T14:32:05.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.833+0000 7fe0eaffd640 10 monclient: get_auth_request con 0x7fe0d806e110 auth_method 0 2024-03-25T14:32:05.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.833+0000 7fe0d37fe640 10 client.4409.objecter ms_handle_connect 0x7fe0d806e110 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fe0e0054760 con 0x7fe0ec06d720 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 10 client.4409.objecter ms_dispatch 0x7fe0ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0e0050270 con 0x7fe0ec06d720 2024-03-25T14:32:05.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:05.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:05.839+0000 7fe0d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:06.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.081+0000 7fe0f14c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:06.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.081+0000 7fe0f14c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:06.086 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.081+0000 7fe0f14c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:06.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.081+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0ec0627b0 con 0x7fe0ec06d720 2024-03-25T14:32:06.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.193+0000 7fe0d37fe640 1 -- 172.21.15.26:0/1261341811 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fe0e00521a0 con 0x7fe0ec06d720 2024-03-25T14:32:06.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.193+0000 7fe0d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:06.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.193+0000 7fe0d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:06.198 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:06.198 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:06.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.195+0000 7fe0f14c9640 10 librados: watch_flush enter 2024-03-25T14:32:06.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 10 librados: watch_flush exit 2024-03-25T14:32:06.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-25T14:32:06.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 10 client.4409.objecter successfully canceled tick 2024-03-25T14:32:06.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 >> v1:172.21.15.121:6800/288813066 conn(0x7fe0d806e110 legacy=0x7fe0d80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:06.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 10 monclient: shutdown 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 >> v1:172.21.15.26:6790/0 conn(0x7fe0ec06d720 legacy=0x7fe0ec1c2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 shutdown_connections 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 >> 172.21.15.26:0/1261341811 conn(0x7fe0ec0b5d10 msgr2=0x7fe0ec1cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 shutdown_connections 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 -- 172.21.15.26:0/1261341811 wait complete. 2024-03-25T14:32:06.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:06.196+0000 7fe0f14c9640 1 librados: shutdown 2024-03-25T14:32:06.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:13.020 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:13.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.129+0000 7f824c08d640 10 monclient: get_monmap_and_config 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.129+0000 7f824c08d640 10 monclient: build_initial_monmap 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.129+0000 7f824c08d640 10 monclient: monmap: 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:13.130614+0000 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:13.130614+0000 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:13.135 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.130+0000 7f824c08d640 1 Processor -- start 2024-03-25T14:32:13.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.130+0000 7f824c08d640 1 -- start start 2024-03-25T14:32:13.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.130+0000 7f824c08d640 10 monclient: init 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient(hunting): picked mon.noname-c con 0x7f82440a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient(hunting): picked mon.noname-b con 0x7f824413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient(hunting): picked mon.noname-a con 0x7f82441430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8244142520 con 0x7f82441430c0 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82440b28a0 con 0x7f82440a98f0 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82440b2a30 con 0x7f824413f710 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:13.137 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.131+0000 7f824c08d640 10 monclient(hunting): authenticate will time out at 1818.234572s 2024-03-25T14:32:13.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.143+0000 7f8249628640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f824413f710 0x7f82441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56542/0 (socket says 172.21.15.26:56542) 2024-03-25T14:32:13.148 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.143+0000 7f8249628640 1 -- 172.21.15.26:0/2673275277 learned_addr learned my addr 172.21.15.26:0/2673275277 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:13.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.151+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225192777 0 0) 0x7f82440b28a0 con 0x7f82440a98f0 2024-03-25T14:32:13.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.151+0000 7f8248e27640 10 monclient(hunting): _negotiate 2024-03-25T14:32:13.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.151+0000 7f8248e27640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:13.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.151+0000 7f8248e27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:13.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.151+0000 7f8248e27640 10 monclient(hunting): my global_id is 4447 2024-03-25T14:32:13.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.151+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8230003520 con 0x7f82440a98f0 2024-03-25T14:32:13.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.157+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404107227 0 0) 0x7f82440b2a30 con 0x7f824413f710 2024-03-25T14:32:13.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.157+0000 7f8248e27640 10 monclient(hunting): _negotiate 2024-03-25T14:32:13.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.157+0000 7f8248e27640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:13.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.157+0000 7f8248e27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:13.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.157+0000 7f8248e27640 10 monclient(hunting): my global_id is 4412 2024-03-25T14:32:13.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.157+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82440b28a0 con 0x7f824413f710 2024-03-25T14:32:13.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.158+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753772313 0 0) 0x7f8244142520 con 0x7f82441430c0 2024-03-25T14:32:13.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.158+0000 7f8248e27640 10 monclient(hunting): _negotiate 2024-03-25T14:32:13.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.158+0000 7f8248e27640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:13.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.158+0000 7f8248e27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:13.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.158+0000 7f8248e27640 10 monclient(hunting): my global_id is 4458 2024-03-25T14:32:13.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.158+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82440b2a30 con 0x7f82441430c0 2024-03-25T14:32:13.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3129648180 0 0) 0x7f8230003520 con 0x7f82440a98f0 2024-03-25T14:32:13.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 >> v1:172.21.15.26:6790/0 conn(0x7f824413f710 legacy=0x7f82441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 >> v1:172.21.15.26:6789/0 conn(0x7f82441430c0 legacy=0x7f824414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: _finish_hunting 0 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 1 monclient: found mon.noname-c 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440b44c0 con 0x7f82440a98f0 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: _finish_auth 0 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: _check_auth_tickets 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8240001690 con 0x7f82440a98f0 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f8248e27640 10 monclient: dump: 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.165 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f824c08d640 5 monclient: authenticate success, global_id 4447 2024-03-25T14:32:13.166 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.159+0000 7f824c08d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82400017f0 con 0x7f82440a98f0 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f8248e27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f8248e27640 1 -- 172.21.15.26:0/2673275277 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8240001c00 con 0x7f82440a98f0 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f824c08d640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f824c08d640 4 set_mon_vals no callback set 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f8248e27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f8248e27640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:13.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.183+0000 7f8248e27640 10 monclient: dump: 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:13.189 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 10 monclient: shutdown 2024-03-25T14:32:13.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:13.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 1 -- 172.21.15.26:0/2673275277 >> v1:172.21.15.121:6789/0 conn(0x7f82440a98f0 legacy=0x7f82440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 1 -- 172.21.15.26:0/2673275277 shutdown_connections 2024-03-25T14:32:13.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 1 -- 172.21.15.26:0/2673275277 >> 172.21.15.26:0/2673275277 conn(0x7f82440b7ea0 msgr2=0x7f82440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:13.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 1 -- 172.21.15.26:0/2673275277 shutdown_connections 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.184+0000 7f824c08d640 1 -- 172.21.15.26:0/2673275277 wait complete. 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 10 monclient: build_initial_monmap 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 10 monclient: monmap: 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 1 librados: starting msgr at 2024-03-25T14:32:13.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 1 librados: starting objecter 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 1 Processor -- start 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 1 -- start start 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 1 librados: setting wanted keys 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 1 librados: calling monclient init 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.185+0000 7f824c08d640 10 monclient: init 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient(hunting): picked mon.noname-b con 0x7f82440a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient(hunting): picked mon.noname-c con 0x7f824413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient(hunting): picked mon.noname-a con 0x7f82441430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8244139870 con 0x7f82441430c0 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82441d51f0 con 0x7f82440a98f0 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8244138830 con 0x7f824413f710 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:13.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.186+0000 7f824c08d640 10 monclient(hunting): authenticate will time out at 1818.288756s 2024-03-25T14:32:13.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.189+0000 7f824a62a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f82441430c0 0x7f82441e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60592/0 (socket says 172.21.15.26:60592) 2024-03-25T14:32:13.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.189+0000 7f824a62a640 1 -- 172.21.15.26:0/3708244125 learned_addr learned my addr 172.21.15.26:0/3708244125 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.195+0000 7f823a7fc640 10 client.?.objecter ms_handle_connect 0x7f82441430c0 2024-03-25T14:32:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.195+0000 7f823a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:13.204 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.199+0000 7f823a7fc640 10 client.?.objecter ms_handle_connect 0x7f82440a98f0 2024-03-25T14:32:13.204 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.199+0000 7f823a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:13.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.201+0000 7f823a7fc640 10 client.?.objecter ms_handle_connect 0x7f824413f710 2024-03-25T14:32:13.206 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.201+0000 7f823a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:13.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.206+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767941518 0 0) 0x7f82441d51f0 con 0x7f82440a98f0 2024-03-25T14:32:13.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.206+0000 7f823a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:13.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.206+0000 7f823a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:13.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.206+0000 7f823a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:13.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.206+0000 7f823a7fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T14:32:13.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.206+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8228004eb0 con 0x7f82440a98f0 2024-03-25T14:32:13.214 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.209+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085223997 0 0) 0x7f8244139870 con 0x7f82441430c0 2024-03-25T14:32:13.214 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82441d51f0 con 0x7f82441430c0 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930896605 0 0) 0x7f8244138830 con 0x7f824413f710 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:13.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:13.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T14:32:13.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.210+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8244139870 con 0x7f824413f710 2024-03-25T14:32:13.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145565431 0 0) 0x7f8228004eb0 con 0x7f82440a98f0 2024-03-25T14:32:13.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8244138830 con 0x7f82440a98f0 2024-03-25T14:32:13.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8240001b00 con 0x7f82440a98f0 2024-03-25T14:32:13.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:13.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 10 monclient(hunting): dump: 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.219+0000 7f823a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:13.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.227+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232565712 0 0) 0x7f8244139870 con 0x7f824413f710 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.227+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8228004eb0 con 0x7f824413f710 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.227+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8234001930 con 0x7f824413f710 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.227+0000 7f823a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.227+0000 7f823a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.227+0000 7f823a7fc640 10 monclient(hunting): dump: 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:13.232 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:13.233 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:13.233 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:13.233 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759753469 0 0) 0x7f8244138830 con 0x7f82440a98f0 2024-03-25T14:32:13.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 >> v1:172.21.15.26:6790/0 conn(0x7f824413f710 legacy=0x7f82441d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 >> v1:172.21.15.26:6789/0 conn(0x7f82441430c0 legacy=0x7f82441e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 1 monclient: found mon.b 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82441d77a0 con 0x7f82440a98f0 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f823a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 5 monclient: authenticate success, global_id 4450 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 10 monclient: _renew_subs 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:13.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82441d83d0 con 0x7f82440a98f0 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 10 monclient: _renew_subs 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8244138c80 con 0x7f82440a98f0 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.230+0000 7f824c08d640 1 librados: init done 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.231+0000 7f824c08d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:13.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.231+0000 7f824c08d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:13.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.231+0000 7f824c08d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:13.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.231+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82440b3e80 con 0x7f82440a98f0 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82400038a0 con 0x7f82440a98f0 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8240003bb0 con 0x7f82440a98f0 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823a7fc640 10 monclient: dump: 2024-03-25T14:32:13.514 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:13.517 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.509+0000 7f823bfff640 4 set_mon_vals no callback set 2024-03-25T14:32:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.510+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f8240019b80 con 0x7f82440a98f0 2024-03-25T14:32:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.510+0000 7f8249628640 10 monclient: get_auth_request con 0x7f822806df70 auth_method 0 2024-03-25T14:32:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.511+0000 7f823a7fc640 10 client.4450.objecter ms_handle_connect 0x7f822806df70 2024-03-25T14:32:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f8240053680 con 0x7f82440a98f0 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 10 client.4450.objecter ms_dispatch 0x7f8244000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f824004efc0 con 0x7f82440a98f0 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.516+0000 7f823a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:13.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.775+0000 7f824c08d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:13.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.775+0000 7f824c08d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:13.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.775+0000 7f824c08d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:13.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.775+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8244074cc0 con 0x7f82440a98f0 2024-03-25T14:32:13.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.864+0000 7f823a7fc640 1 -- 172.21.15.26:0/3708244125 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f82400510c0 con 0x7f82440a98f0 2024-03-25T14:32:13.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.864+0000 7f823a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:13.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.864+0000 7f823a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:13.966 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:13.966 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 10 librados: watch_flush enter 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 10 librados: watch_flush exit 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 10 client.4450.objecter successfully canceled tick 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 >> v1:172.21.15.121:6800/288813066 conn(0x7f822806df70 legacy=0x7f8228070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 10 monclient: shutdown 2024-03-25T14:32:13.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:13.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 >> v1:172.21.15.121:6789/0 conn(0x7f82440a98f0 legacy=0x7f8244138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:13.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 shutdown_connections 2024-03-25T14:32:13.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 >> 172.21.15.26:0/3708244125 conn(0x7f82440b7ea0 msgr2=0x7f8244074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:13.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 shutdown_connections 2024-03-25T14:32:13.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 -- 172.21.15.26:0/3708244125 wait complete. 2024-03-25T14:32:13.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:13.867+0000 7f824c08d640 1 librados: shutdown 2024-03-25T14:32:13.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:16.402 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:35.263 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.413 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:35.273 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.452 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:35.328 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.463 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:35.324 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.465 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:35.341 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.487 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:35.348 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.490 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:35.367 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.498 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:35.359 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:16.528 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:35.389 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:20.777 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:20.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.879+0000 7f5730f88640 10 monclient: get_monmap_and_config 2024-03-25T14:32:20.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.879+0000 7f5730f88640 10 monclient: build_initial_monmap 2024-03-25T14:32:20.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.879+0000 7f5730f88640 10 monclient: monmap: 2024-03-25T14:32:20.884 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:20.884 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:20.880382+0000 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:20.880382+0000 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:20.885 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.880+0000 7f5730f88640 1 Processor -- start 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.880+0000 7f5730f88640 1 -- start start 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.880+0000 7f5730f88640 10 monclient: init 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient(hunting): picked mon.noname-c con 0x7f572c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient(hunting): picked mon.noname-b con 0x7f572c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient(hunting): picked mon.noname-a con 0x7f572c1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c142520 con 0x7f572c1430c0 2024-03-25T14:32:20.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c0b28a0 con 0x7f572c0a98f0 2024-03-25T14:32:20.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c0b2a30 con 0x7f572c13f710 2024-03-25T14:32:20.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:20.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.881+0000 7f5730f88640 10 monclient(hunting): authenticate will time out at 1825.984134s 2024-03-25T14:32:20.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.887+0000 7f5729d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f572c13f710 0x7f572c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58376/0 (socket says 172.21.15.26:58376) 2024-03-25T14:32:20.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.887+0000 7f5729d9b640 1 -- 172.21.15.26:0/658785185 learned_addr learned my addr 172.21.15.26:0/658785185 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:20.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.901+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429075407 0 0) 0x7f572c0b2a30 con 0x7f572c13f710 2024-03-25T14:32:20.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.901+0000 7f572959a640 10 monclient(hunting): _negotiate 2024-03-25T14:32:20.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.901+0000 7f572959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:20.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.901+0000 7f572959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:20.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.901+0000 7f572959a640 10 monclient(hunting): my global_id is 4418 2024-03-25T14:32:20.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.901+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5714003520 con 0x7f572c13f710 2024-03-25T14:32:20.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823477723 0 0) 0x7f5714003520 con 0x7f572c13f710 2024-03-25T14:32:20.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 >> v1:172.21.15.121:6789/0 conn(0x7f572c0a98f0 legacy=0x7f572c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:32:20.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 >> v1:172.21.15.26:6789/0 conn(0x7f572c1430c0 legacy=0x7f572c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:20.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: _finish_hunting 0 2024-03-25T14:32:20.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 1 monclient: found mon.noname-b 2024-03-25T14:32:20.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0b44c0 con 0x7f572c13f710 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: _finish_auth 0 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: _check_auth_tickets 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f5730f88640 5 monclient: authenticate success, global_id 4418 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f5730f88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5720001690 con 0x7f572c13f710 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:32:20.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f572959a640 10 monclient: dump: 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:20.924 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:20.925 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:20.925 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:20.925 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.917+0000 7f5730f88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:20.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5720001920 con 0x7f572c13f710 2024-03-25T14:32:20.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f572959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:20.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f572959a640 1 -- 172.21.15.26:0/658785185 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5720001cf0 con 0x7f572c13f710 2024-03-25T14:32:20.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f5730f88640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f5730f88640 4 set_mon_vals no callback set 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f572959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f572959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.924+0000 7f572959a640 10 monclient: dump: 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:20.930 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:20.931 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:20.931 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:20.931 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 10 monclient: shutdown 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 1 -- 172.21.15.26:0/658785185 >> v1:172.21.15.26:6790/0 conn(0x7f572c13f710 legacy=0x7f572c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 1 -- 172.21.15.26:0/658785185 shutdown_connections 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 1 -- 172.21.15.26:0/658785185 >> 172.21.15.26:0/658785185 conn(0x7f572c0b7ea0 msgr2=0x7f572c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 1 -- 172.21.15.26:0/658785185 shutdown_connections 2024-03-25T14:32:20.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.925+0000 7f5730f88640 1 -- 172.21.15.26:0/658785185 wait complete. 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.926+0000 7f5730f88640 10 monclient: build_initial_monmap 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.926+0000 7f5730f88640 10 monclient: monmap: 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:20.934 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 1 librados: starting msgr at 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 1 librados: starting objecter 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 1 Processor -- start 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 1 -- start start 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 1 librados: setting wanted keys 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 1 librados: calling monclient init 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.927+0000 7f5730f88640 10 monclient: init 2024-03-25T14:32:20.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient(hunting): picked mon.noname-a con 0x7f572c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient(hunting): picked mon.noname-c con 0x7f572c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient(hunting): picked mon.noname-b con 0x7f572c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c1385c0 con 0x7f572c0a98f0 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c138ae0 con 0x7f572c1430c0 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c0b1660 con 0x7f572c13f710 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:20.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.928+0000 7f5730f88640 10 monclient(hunting): authenticate will time out at 1826.030752s 2024-03-25T14:32:20.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.929+0000 7f572ad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f572c1430c0 0x7f572c1e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:58524/0 (socket says 172.21.15.26:58524) 2024-03-25T14:32:20.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.929+0000 7f572ad9d640 1 -- 172.21.15.26:0/2606259708 learned_addr learned my addr 172.21.15.26:0/2606259708 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:20.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.930+0000 7f571affd640 10 client.?.objecter ms_handle_connect 0x7f572c1430c0 2024-03-25T14:32:20.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.930+0000 7f571affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:20.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.939+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700970906 0 0) 0x7f572c138ae0 con 0x7f572c1430c0 2024-03-25T14:32:20.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.939+0000 7f571affd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:20.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.939+0000 7f571affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:20.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.939+0000 7f571affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:20.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.939+0000 7f571affd640 10 monclient(hunting): my global_id is 4453 2024-03-25T14:32:20.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.939+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c004eb0 con 0x7f572c1430c0 2024-03-25T14:32:20.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.942+0000 7f571affd640 10 client.?.objecter ms_handle_connect 0x7f572c0a98f0 2024-03-25T14:32:20.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.942+0000 7f571affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:20.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.943+0000 7f571affd640 10 client.?.objecter ms_handle_connect 0x7f572c13f710 2024-03-25T14:32:20.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.943+0000 7f571affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:20.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.947+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176019442 0 0) 0x7f572c0b1660 con 0x7f572c13f710 2024-03-25T14:32:20.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.947+0000 7f571affd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:20.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.947+0000 7f571affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:20.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.947+0000 7f571affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:20.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.947+0000 7f571affd640 10 monclient(hunting): my global_id is 4421 2024-03-25T14:32:20.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.947+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572c138ae0 con 0x7f572c13f710 2024-03-25T14:32:20.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.949+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168289407 0 0) 0x7f570c004eb0 con 0x7f572c1430c0 2024-03-25T14:32:20.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.949+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f572c0b1660 con 0x7f572c1430c0 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.950+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5724001cf0 con 0x7f572c1430c0 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.950+0000 7f571affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.950+0000 7f571affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.950+0000 7f571affd640 10 monclient(hunting): dump: 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:20.955 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:20.956 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:20.956 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:20.956 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:20.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.950+0000 7f571affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:20.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.953+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422889647 0 0) 0x7f572c1385c0 con 0x7f572c0a98f0 2024-03-25T14:32:20.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.954+0000 7f571affd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:20.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.954+0000 7f571affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:20.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.954+0000 7f571affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:20.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.954+0000 7f571affd640 10 monclient(hunting): my global_id is 4467 2024-03-25T14:32:20.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.954+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c004eb0 con 0x7f572c0a98f0 2024-03-25T14:32:20.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2899529063 0 0) 0x7f572c0b1660 con 0x7f572c1430c0 2024-03-25T14:32:20.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 >> v1:172.21.15.26:6790/0 conn(0x7f572c13f710 legacy=0x7f572c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:20.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 >> v1:172.21.15.26:6789/0 conn(0x7f572c0a98f0 legacy=0x7f572c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:20.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 10 monclient: _finish_hunting 0 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 1 monclient: found mon.b 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c137fa0 con 0x7f572c1430c0 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 10 monclient: _finish_auth 0 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 10 monclient: _check_auth_tickets 2024-03-25T14:32:20.961 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f571affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 5 monclient: authenticate success, global_id 4453 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 10 monclient: _renew_subs 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c1381f0 con 0x7f572c1430c0 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 10 monclient: _renew_subs 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c1db170 con 0x7f572c1430c0 2024-03-25T14:32:20.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.955+0000 7f5730f88640 1 librados: init done 2024-03-25T14:32:20.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.956+0000 7f5730f88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:20.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.956+0000 7f5730f88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:20.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.956+0000 7f5730f88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:20.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:20.956+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c14f2d0 con 0x7f572c1430c0 2024-03-25T14:32:21.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5724003990 con 0x7f572c1430c0 2024-03-25T14:32:21.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f571affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:21.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5724003da0 con 0x7f572c1430c0 2024-03-25T14:32:21.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f571affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f571affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f571affd640 10 monclient: dump: 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:21.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.235+0000 7f5728d99640 4 set_mon_vals no callback set 2024-03-25T14:32:21.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.236+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f5724003fc0 con 0x7f572c1430c0 2024-03-25T14:32:21.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.237+0000 7f572a59c640 10 monclient: get_auth_request con 0x7f570c06de00 auth_method 0 2024-03-25T14:32:21.242 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.237+0000 7f571affd640 10 client.4453.objecter ms_handle_connect 0x7f570c06de00 2024-03-25T14:32:21.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f5724054ae0 con 0x7f572c1430c0 2024-03-25T14:32:21.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 10 client.4453.objecter ms_dispatch 0x7f572c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:21.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:21.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:21.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T14:32:21.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57240505f0 con 0x7f572c1430c0 2024-03-25T14:32:21.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:21.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.241+0000 7f571affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:21.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.492+0000 7f5730f88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:21.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.492+0000 7f5730f88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:21.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.492+0000 7f5730f88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:21.497 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.492+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f572c0627b0 con 0x7f572c1430c0 2024-03-25T14:32:21.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.517+0000 7f571affd640 1 -- 172.21.15.26:0/2606259708 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f5724052520 con 0x7f572c1430c0 2024-03-25T14:32:21.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.517+0000 7f571affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:21.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.517+0000 7f571affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:21.522 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:21.522 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:21.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 10 librados: watch_flush enter 2024-03-25T14:32:21.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 10 librados: watch_flush exit 2024-03-25T14:32:21.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T14:32:21.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 10 client.4453.objecter successfully canceled tick 2024-03-25T14:32:21.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 >> v1:172.21.15.121:6800/288813066 conn(0x7f570c06de00 legacy=0x7f570c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 10 monclient: shutdown 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.519+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 >> v1:172.21.15.121:6789/0 conn(0x7f572c1430c0 legacy=0x7f572c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.520+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 shutdown_connections 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.520+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 >> 172.21.15.26:0/2606259708 conn(0x7f572c0b7ea0 msgr2=0x7f572c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.520+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 shutdown_connections 2024-03-25T14:32:21.526 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.520+0000 7f5730f88640 1 -- 172.21.15.26:0/2606259708 wait complete. 2024-03-25T14:32:21.527 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:21.520+0000 7f5730f88640 1 librados: shutdown 2024-03-25T14:32:21.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:28.344 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:28.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.454+0000 7fada1f0e640 10 monclient: get_monmap_and_config 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.454+0000 7fada1f0e640 10 monclient: build_initial_monmap 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.454+0000 7fada1f0e640 10 monclient: monmap: 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:28.455711+0000 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:28.455711+0000 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:28.460 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.455+0000 7fada1f0e640 1 Processor -- start 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.455+0000 7fada1f0e640 1 -- start start 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.455+0000 7fada1f0e640 10 monclient: init 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient(hunting): picked mon.noname-c con 0x7fad9c13fc60 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient(hunting): picked mon.noname-a con 0x7fad9c070570 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient(hunting): picked mon.noname-b con 0x7fad9c0727a0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c13c9f0 con 0x7fad9c070570 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c06fac0 con 0x7fad9c13fc60 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c0715d0 con 0x7fad9c0727a0 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:28.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.456+0000 7fada1f0e640 10 monclient(hunting): authenticate will time out at 1833.559260s 2024-03-25T14:32:28.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.468+0000 7fad9affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fad9c070570 0x7fad9c072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35598/0 (socket says 172.21.15.26:35598) 2024-03-25T14:32:28.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.468+0000 7fad9affd640 1 -- 172.21.15.26:0/1652165711 learned_addr learned my addr 172.21.15.26:0/1652165711 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:28.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.477+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004930893 0 0) 0x7fad9c0715d0 con 0x7fad9c0727a0 2024-03-25T14:32:28.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.477+0000 7fad9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:28.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.477+0000 7fad9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:28.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.477+0000 7fad9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:28.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.477+0000 7fad9a7fc640 10 monclient(hunting): my global_id is 4424 2024-03-25T14:32:28.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.477+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad84003520 con 0x7fad9c0727a0 2024-03-25T14:32:28.489 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1723698152 0 0) 0x7fad84003520 con 0x7fad9c0727a0 2024-03-25T14:32:28.489 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 >> v1:172.21.15.121:6789/0 conn(0x7fad9c13fc60 legacy=0x7fad9c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:28.489 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 >> v1:172.21.15.26:6789/0 conn(0x7fad9c070570 legacy=0x7fad9c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:28.489 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 1 monclient: found mon.noname-b 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c0718b0 con 0x7fad9c0727a0 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fad94001a10 con 0x7fad9c0727a0 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fad9a7fc640 10 monclient: dump: 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.490 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fada1f0e640 5 monclient: authenticate success, global_id 4424 2024-03-25T14:32:28.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.484+0000 7fada1f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad94001b70 con 0x7fad9c0727a0 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fad9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fad9a7fc640 1 -- 172.21.15.26:0/1652165711 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fad94001fc0 con 0x7fad9c0727a0 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fada1f0e640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fada1f0e640 4 set_mon_vals no callback set 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fad9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:28.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fad9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.493+0000 7fad9a7fc640 10 monclient: dump: 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:28.499 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 10 monclient: shutdown 2024-03-25T14:32:28.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:28.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 1 -- 172.21.15.26:0/1652165711 >> v1:172.21.15.26:6790/0 conn(0x7fad9c0727a0 legacy=0x7fad9c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:28.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 1 -- 172.21.15.26:0/1652165711 shutdown_connections 2024-03-25T14:32:28.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 1 -- 172.21.15.26:0/1652165711 >> 172.21.15.26:0/1652165711 conn(0x7fad9c0b4a80 msgr2=0x7fad9c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 1 -- 172.21.15.26:0/1652165711 shutdown_connections 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.494+0000 7fada1f0e640 1 -- 172.21.15.26:0/1652165711 wait complete. 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.495+0000 7fada1f0e640 10 monclient: build_initial_monmap 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.495+0000 7fada1f0e640 10 monclient: monmap: 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:28.502 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.495+0000 7fada1f0e640 1 librados: starting msgr at 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.495+0000 7fada1f0e640 1 librados: starting objecter 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.495+0000 7fada1f0e640 1 Processor -- start 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 1 -- start start 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 1 librados: setting wanted keys 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 1 librados: calling monclient init 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient: init 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient(hunting): picked mon.noname-b con 0x7fad9c070570 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient(hunting): picked mon.noname-c con 0x7fad9c0727a0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient(hunting): picked mon.noname-a con 0x7fad9c13fc60 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c0ad9c0 con 0x7fad9c13fc60 2024-03-25T14:32:28.503 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c0ae3d0 con 0x7fad9c070570 2024-03-25T14:32:28.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c135c70 con 0x7fad9c0727a0 2024-03-25T14:32:28.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:28.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.496+0000 7fada1f0e640 10 monclient(hunting): authenticate will time out at 1833.599260s 2024-03-25T14:32:28.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.508+0000 7fad9b7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fad9c070570 0x7fad9c136370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:53228/0 (socket says 172.21.15.26:53228) 2024-03-25T14:32:28.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.508+0000 7fad9b7fe640 1 -- 172.21.15.26:0/146206291 learned_addr learned my addr 172.21.15.26:0/146206291 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:28.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.509+0000 7fad83fff640 10 client.?.objecter ms_handle_connect 0x7fad9c0727a0 2024-03-25T14:32:28.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.509+0000 7fad83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:28.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.513+0000 7fad83fff640 10 client.?.objecter ms_handle_connect 0x7fad9c070570 2024-03-25T14:32:28.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.513+0000 7fad83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:28.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.523+0000 7fad83fff640 10 client.?.objecter ms_handle_connect 0x7fad9c13fc60 2024-03-25T14:32:28.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.523+0000 7fad83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:28.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.526+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346066464 0 0) 0x7fad9c135c70 con 0x7fad9c0727a0 2024-03-25T14:32:28.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.526+0000 7fad83fff640 10 monclient(hunting): _negotiate 2024-03-25T14:32:28.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.526+0000 7fad83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:28.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.526+0000 7fad83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:28.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.526+0000 7fad83fff640 10 monclient(hunting): my global_id is 4427 2024-03-25T14:32:28.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.526+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad88004eb0 con 0x7fad9c0727a0 2024-03-25T14:32:28.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.527+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419819256 0 0) 0x7fad9c0ad9c0 con 0x7fad9c13fc60 2024-03-25T14:32:28.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.527+0000 7fad83fff640 10 monclient(hunting): _negotiate 2024-03-25T14:32:28.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.527+0000 7fad83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:28.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.527+0000 7fad83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:28.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.527+0000 7fad83fff640 10 monclient(hunting): my global_id is 4473 2024-03-25T14:32:28.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.527+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9c135c70 con 0x7fad9c13fc60 2024-03-25T14:32:28.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.528+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556421629 0 0) 0x7fad9c0ae3d0 con 0x7fad9c070570 2024-03-25T14:32:28.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.528+0000 7fad83fff640 10 monclient(hunting): _negotiate 2024-03-25T14:32:28.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.528+0000 7fad83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:28.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.528+0000 7fad83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:28.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.528+0000 7fad83fff640 10 monclient(hunting): my global_id is 4456 2024-03-25T14:32:28.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.528+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9c0ad9c0 con 0x7fad9c070570 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534582339 0 0) 0x7fad9c135c70 con 0x7fad9c13fc60 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad9c0ae3d0 con 0x7fad9c13fc60 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fad94001e80 con 0x7fad9c13fc60 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 10 monclient(hunting): dump: 2024-03-25T14:32:28.547 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.542+0000 7fad83fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.544+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048372805 0 0) 0x7fad9c0ad9c0 con 0x7fad9c070570 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.544+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad9c135c70 con 0x7fad9c070570 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.544+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fad8c0018d0 con 0x7fad9c070570 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.544+0000 7fad83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.544+0000 7fad83fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.544+0000 7fad83fff640 10 monclient(hunting): dump: 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.549 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.546+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3221711098 0 0) 0x7fad88004eb0 con 0x7fad9c0727a0 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.546+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad9c0ad9c0 con 0x7fad9c0727a0 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.546+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fad900018a0 con 0x7fad9c0727a0 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.546+0000 7fad83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.546+0000 7fad83fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.546+0000 7fad83fff640 10 monclient(hunting): dump: 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:28.551 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:28.552 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:28.552 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.558 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 762330728 0 0) 0x7fad9c0ad9c0 con 0x7fad9c0727a0 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 >> v1:172.21.15.121:6789/0 conn(0x7fad9c070570 legacy=0x7fad9c136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 >> v1:172.21.15.26:6789/0 conn(0x7fad9c13fc60 legacy=0x7fad9c0ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 10 monclient: _finish_hunting 0 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 1 monclient: found mon.c 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c135f50 con 0x7fad9c0727a0 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 10 monclient: _finish_auth 0 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 10 monclient: _check_auth_tickets 2024-03-25T14:32:28.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fad83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 5 monclient: authenticate success, global_id 4427 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 10 monclient: _renew_subs 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9c0ac610 con 0x7fad9c0727a0 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 10 monclient: _renew_subs 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9c1e13d0 con 0x7fad9c0727a0 2024-03-25T14:32:28.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.553+0000 7fada1f0e640 1 librados: init done 2024-03-25T14:32:28.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.555+0000 7fada1f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:28.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.555+0000 7fada1f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:28.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.555+0000 7fada1f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:28.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.555+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9c13b950 con 0x7fad9c0727a0 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad900047e0 con 0x7fad9c0727a0 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fad90004950 con 0x7fad9c0727a0 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 10 monclient: dump: 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.877 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:28.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad99ffb640 4 set_mon_vals no callback set 2024-03-25T14:32:28.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.872+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fad9001a920 con 0x7fad9c0727a0 2024-03-25T14:32:28.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.873+0000 7fad9b7fe640 10 monclient: get_auth_request con 0x7fad8806e110 auth_method 0 2024-03-25T14:32:28.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.873+0000 7fad83fff640 10 client.4427.objecter ms_handle_connect 0x7fad8806e110 2024-03-25T14:32:28.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fad900545e0 con 0x7fad9c0727a0 2024-03-25T14:32:28.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 10 client.4427.objecter ms_dispatch 0x7fad9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:28.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:28.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:28.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-25T14:32:28.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad9004ff20 con 0x7fad9c0727a0 2024-03-25T14:32:28.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:28.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:28.881+0000 7fad83fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:29.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.131+0000 7fada1f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:29.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.131+0000 7fada1f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:29.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.131+0000 7fada1f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:29.136 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.131+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad9c0627b0 con 0x7fad9c0727a0 2024-03-25T14:32:29.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.249+0000 7fad83fff640 1 -- 172.21.15.26:0/146206291 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fad90052020 con 0x7fad9c0727a0 2024-03-25T14:32:29.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.249+0000 7fad83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:29.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.249+0000 7fad83fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:29.255 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:29.255 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:29.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 10 librados: watch_flush enter 2024-03-25T14:32:29.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 10 librados: watch_flush exit 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 10 client.4427.objecter successfully canceled tick 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 >> v1:172.21.15.121:6800/288813066 conn(0x7fad8806e110 legacy=0x7fad880705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 10 monclient: shutdown 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 >> v1:172.21.15.26:6790/0 conn(0x7fad9c0727a0 legacy=0x7fad9c0aacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 shutdown_connections 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 >> 172.21.15.26:0/146206291 conn(0x7fad9c0b4a80 msgr2=0x7fad9c13bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 shutdown_connections 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 -- 172.21.15.26:0/146206291 wait complete. 2024-03-25T14:32:29.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:29.252+0000 7fada1f0e640 1 librados: shutdown 2024-03-25T14:32:29.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:36.076 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.188+0000 7f1aa869d640 10 monclient: get_monmap_and_config 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.188+0000 7f1aa869d640 10 monclient: build_initial_monmap 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.188+0000 7f1aa869d640 10 monclient: monmap: 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:36.189128+0000 2024-03-25T14:32:36.193 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:36.189128+0000 2024-03-25T14:32:36.194 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:36.194 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:36.194 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:36.194 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:36.194 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:36.194 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:36.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 1 Processor -- start 2024-03-25T14:32:36.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 1 -- start start 2024-03-25T14:32:36.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient: init 2024-03-25T14:32:36.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient(hunting): picked mon.noname-c con 0x7f1aa00a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient(hunting): picked mon.noname-a con 0x7f1aa013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient(hunting): picked mon.noname-b con 0x7f1aa01430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa0142520 con 0x7f1aa013f710 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00b28a0 con 0x7f1aa00a98f0 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00b2a30 con 0x7f1aa01430c0 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:36.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.189+0000 7f1aa869d640 10 monclient(hunting): authenticate will time out at 1841.292592s 2024-03-25T14:32:36.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.204+0000 7f1aa6439640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1aa00a98f0 0x7f1aa00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:57902/0 (socket says 172.21.15.26:57902) 2024-03-25T14:32:36.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.204+0000 7f1aa6439640 1 -- 172.21.15.26:0/3761730721 learned_addr learned my addr 172.21.15.26:0/3761730721 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:36.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.216+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914094225 0 0) 0x7f1aa00b28a0 con 0x7f1aa00a98f0 2024-03-25T14:32:36.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.216+0000 7f1aa5437640 10 monclient(hunting): _negotiate 2024-03-25T14:32:36.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.216+0000 7f1aa5437640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:36.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.216+0000 7f1aa5437640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:36.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.216+0000 7f1aa5437640 10 monclient(hunting): my global_id is 4459 2024-03-25T14:32:36.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.216+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8c003520 con 0x7f1aa00a98f0 2024-03-25T14:32:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.221+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435143616 0 0) 0x7f1aa00b2a30 con 0x7f1aa01430c0 2024-03-25T14:32:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.221+0000 7f1aa5437640 10 monclient(hunting): _negotiate 2024-03-25T14:32:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.221+0000 7f1aa5437640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.221+0000 7f1aa5437640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.221+0000 7f1aa5437640 10 monclient(hunting): my global_id is 4430 2024-03-25T14:32:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.221+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa00b28a0 con 0x7f1aa01430c0 2024-03-25T14:32:36.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783639634 0 0) 0x7f1aa00b28a0 con 0x7f1aa01430c0 2024-03-25T14:32:36.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 >> v1:172.21.15.121:6789/0 conn(0x7f1aa00a98f0 legacy=0x7f1aa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:36.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 >> v1:172.21.15.26:6789/0 conn(0x7f1aa013f710 legacy=0x7f1aa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:36.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 10 monclient: _finish_hunting 0 2024-03-25T14:32:36.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 1 monclient: found mon.noname-b 2024-03-25T14:32:36.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00b44c0 con 0x7f1aa01430c0 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 10 monclient: _finish_auth 0 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 10 monclient: _check_auth_tickets 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa5437640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa869d640 5 monclient: authenticate success, global_id 4430 2024-03-25T14:32:36.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.230+0000 7f1aa869d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.231+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1a980019b0 con 0x7f1aa01430c0 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.231+0000 7f1aa5437640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.231+0000 7f1aa5437640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.231+0000 7f1aa5437640 10 monclient: dump: 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:36.239 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:36.240 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:36.240 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:36.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.231+0000 7f1aa869d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:36.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.243+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a98001c40 con 0x7f1aa01430c0 2024-03-25T14:32:36.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.243+0000 7f1aa5437640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.243+0000 7f1aa5437640 1 -- 172.21.15.26:0/3761730721 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1a98001f50 con 0x7f1aa01430c0 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.243+0000 7f1aa5437640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa5437640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa5437640 10 monclient: dump: 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:36.249 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:36.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:36.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 4 set_mon_vals no callback set 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 10 monclient: shutdown 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 1 -- 172.21.15.26:0/3761730721 >> v1:172.21.15.26:6790/0 conn(0x7f1aa01430c0 legacy=0x7f1aa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 1 -- 172.21.15.26:0/3761730721 shutdown_connections 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 1 -- 172.21.15.26:0/3761730721 >> 172.21.15.26:0/3761730721 conn(0x7f1aa00b7ea0 msgr2=0x7f1aa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 1 -- 172.21.15.26:0/3761730721 shutdown_connections 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.244+0000 7f1aa869d640 1 -- 172.21.15.26:0/3761730721 wait complete. 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.245+0000 7f1aa869d640 10 monclient: build_initial_monmap 2024-03-25T14:32:36.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.245+0000 7f1aa869d640 10 monclient: monmap: 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 1 librados: starting msgr at 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 1 librados: starting objecter 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 1 Processor -- start 2024-03-25T14:32:36.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 1 -- start start 2024-03-25T14:32:36.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 1 librados: setting wanted keys 2024-03-25T14:32:36.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 1 librados: calling monclient init 2024-03-25T14:32:36.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.246+0000 7f1aa869d640 10 monclient: init 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient(hunting): picked mon.noname-c con 0x7f1aa00a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient(hunting): picked mon.noname-b con 0x7f1aa013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient(hunting): picked mon.noname-a con 0x7f1aa01430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa01d51f0 con 0x7f1aa01430c0 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00abcc0 con 0x7f1aa013f710 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa01dab50 con 0x7f1aa00a98f0 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:36.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.247+0000 7f1aa869d640 10 monclient(hunting): authenticate will time out at 1841.350298s 2024-03-25T14:32:36.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.255+0000 7f1aa6439640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1aa00a98f0 0x7f1aa0138740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49914/0 (socket says 172.21.15.26:49914) 2024-03-25T14:32:36.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.255+0000 7f1aa6439640 1 -- 172.21.15.26:0/253574469 learned_addr learned my addr 172.21.15.26:0/253574469 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:36.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.258+0000 7f1a96ffd640 10 client.?.objecter ms_handle_connect 0x7f1aa00a98f0 2024-03-25T14:32:36.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.258+0000 7f1a96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:36.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.259+0000 7f1a96ffd640 10 client.?.objecter ms_handle_connect 0x7f1aa01430c0 2024-03-25T14:32:36.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.259+0000 7f1a96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.266+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223320716 0 0) 0x7f1aa01dab50 con 0x7f1aa00a98f0 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.266+0000 7f1a96ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.266+0000 7f1a96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.266+0000 7f1a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.266+0000 7f1a96ffd640 10 monclient(hunting): my global_id is 4433 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.266+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a84004eb0 con 0x7f1aa00a98f0 2024-03-25T14:32:36.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 10 client.?.objecter ms_handle_connect 0x7f1aa013f710 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115083521 0 0) 0x7f1aa01d51f0 con 0x7f1aa01430c0 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T14:32:36.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.267+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa01dab50 con 0x7f1aa01430c0 2024-03-25T14:32:36.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034959667 0 0) 0x7f1aa01dab50 con 0x7f1aa01430c0 2024-03-25T14:32:36.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aa01d51f0 con 0x7f1aa01430c0 2024-03-25T14:32:36.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1a98001f50 con 0x7f1aa01430c0 2024-03-25T14:32:36.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:36.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:32:36.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 10 monclient(hunting): dump: 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:36.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.278+0000 7f1a96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:36.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.283+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185253629 0 0) 0x7f1aa00abcc0 con 0x7f1aa013f710 2024-03-25T14:32:36.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.283+0000 7f1a96ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:36.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.283+0000 7f1a96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:36.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.283+0000 7f1a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:36.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.283+0000 7f1a96ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T14:32:36.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.283+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa01dab50 con 0x7f1aa013f710 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2677524272 0 0) 0x7f1aa01d51f0 con 0x7f1aa01430c0 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 >> v1:172.21.15.26:6790/0 conn(0x7f1aa00a98f0 legacy=0x7f1aa0138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 >> v1:172.21.15.121:6789/0 conn(0x7f1aa013f710 legacy=0x7f1aa01db070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 1 monclient: found mon.a 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa01db780 con 0x7f1aa01430c0 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 10 monclient: _finish_auth 0 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 5 monclient: authenticate success, global_id 4479 2024-03-25T14:32:36.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 10 monclient: _renew_subs 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa01379a0 con 0x7f1aa01430c0 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 10 monclient: _renew_subs 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa0137d40 con 0x7f1aa01430c0 2024-03-25T14:32:36.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.291+0000 7f1aa869d640 1 librados: init done 2024-03-25T14:32:36.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.292+0000 7f1aa869d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:36.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.292+0000 7f1aa869d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:36.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.292+0000 7f1aa869d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:32:36.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.292+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa00627b0 con 0x7f1aa01430c0 2024-03-25T14:32:36.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a980020b0 con 0x7f1aa01430c0 2024-03-25T14:32:36.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:36.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f1a98003c20 con 0x7f1aa01430c0 2024-03-25T14:32:36.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:36.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 10 monclient: dump: 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1aa4c36640 4 set_mon_vals no callback set 2024-03-25T14:32:36.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.695+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f1a98019cb0 con 0x7f1aa01430c0 2024-03-25T14:32:36.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.696+0000 7f1aa6439640 10 monclient: get_auth_request con 0x7f1a8406ddc0 auth_method 0 2024-03-25T14:32:36.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.696+0000 7f1a96ffd640 10 client.4479.objecter ms_handle_connect 0x7f1a8406ddc0 2024-03-25T14:32:36.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f1a98053960 con 0x7f1aa01430c0 2024-03-25T14:32:36.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 10 client.4479.objecter ms_dispatch 0x7f1aa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:36.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:36.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:36.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T14:32:36.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a9804f2a0 con 0x7f1aa01430c0 2024-03-25T14:32:36.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:36.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.703+0000 7f1a96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:36.797 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:55.657 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.804 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:55.665 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.830 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:05:55.690 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.859 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:55.734 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.866 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:55.742 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.875 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:55.735 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.876 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:05:55.753 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.888 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:55.748 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.916 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:05:55.776 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:36.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.917+0000 7f1aa869d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.917+0000 7f1aa869d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.917+0000 7f1aa869d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:32:36.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:36.917+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aa0141c20 con 0x7f1aa01430c0 2024-03-25T14:32:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.014+0000 7f1a96ffd640 1 -- 172.21.15.26:0/253574469 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f1a980513a0 con 0x7f1aa01430c0 2024-03-25T14:32:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.014+0000 7f1a96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:37.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.014+0000 7f1a96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:37.019 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:37.019 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 10 librados: watch_flush enter 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 10 librados: watch_flush exit 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 10 client.4479.objecter successfully canceled tick 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 >> v1:172.21.15.121:6800/288813066 conn(0x7f1a8406ddc0 legacy=0x7f1a84070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 10 monclient: shutdown 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 >> v1:172.21.15.26:6789/0 conn(0x7f1aa01430c0 legacy=0x7f1aa01e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 shutdown_connections 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 >> 172.21.15.26:0/253574469 conn(0x7f1aa00b7ea0 msgr2=0x7f1aa0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:37.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 shutdown_connections 2024-03-25T14:32:37.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 -- 172.21.15.26:0/253574469 wait complete. 2024-03-25T14:32:37.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:37.016+0000 7f1aa869d640 1 librados: shutdown 2024-03-25T14:32:37.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:43.839 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:43.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.950+0000 7f3dd7c86640 10 monclient: get_monmap_and_config 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.950+0000 7f3dd7c86640 10 monclient: build_initial_monmap 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.950+0000 7f3dd7c86640 10 monclient: monmap: 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:43.952301+0000 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:43.952301+0000 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:43.957 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:43.958 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:43.958 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:43.958 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:43.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.951+0000 7f3dd7c86640 1 Processor -- start 2024-03-25T14:32:43.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.951+0000 7f3dd7c86640 1 -- start start 2024-03-25T14:32:43.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.951+0000 7f3dd7c86640 10 monclient: init 2024-03-25T14:32:43.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:43.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:32:43.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient(hunting): picked mon.noname-b con 0x7f3dd006adf0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient(hunting): picked mon.noname-a con 0x7f3dd00af580 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient(hunting): picked mon.noname-c con 0x7f3dd01409d0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd00b0b20 con 0x7f3dd00af580 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd00afce0 con 0x7f3dd01409d0 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd00af9e0 con 0x7f3dd006adf0 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:43.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.952+0000 7f3dd7c86640 10 monclient(hunting): authenticate will time out at 1849.056036s 2024-03-25T14:32:43.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.958+0000 7f3dd6223640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3dd01409d0 0x7f3dd014b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:53518/0 (socket says 172.21.15.26:53518) 2024-03-25T14:32:43.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.958+0000 7f3dd5a22640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3dd006adf0 0x7f3dd013d630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60318/0 (socket says 172.21.15.26:60318) 2024-03-25T14:32:43.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.958+0000 7f3dd6223640 1 -- 172.21.15.26:0/1443566688 learned_addr learned my addr 172.21.15.26:0/1443566688 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:43.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.969+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473017563 0 0) 0x7f3dd00afce0 con 0x7f3dd01409d0 2024-03-25T14:32:43.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.969+0000 7f3dd4a20640 10 monclient(hunting): _negotiate 2024-03-25T14:32:43.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.969+0000 7f3dd4a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:43.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.969+0000 7f3dd4a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:43.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.969+0000 7f3dd4a20640 10 monclient(hunting): my global_id is 4465 2024-03-25T14:32:43.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.969+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc003520 con 0x7f3dd01409d0 2024-03-25T14:32:43.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.970+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710827644 0 0) 0x7f3dd00af9e0 con 0x7f3dd006adf0 2024-03-25T14:32:43.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.970+0000 7f3dd4a20640 10 monclient(hunting): _negotiate 2024-03-25T14:32:43.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.970+0000 7f3dd4a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:43.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.970+0000 7f3dd4a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:43.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.970+0000 7f3dd4a20640 10 monclient(hunting): my global_id is 4436 2024-03-25T14:32:43.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.970+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd00afce0 con 0x7f3dd006adf0 2024-03-25T14:32:43.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911151109 0 0) 0x7f3dbc003520 con 0x7f3dd01409d0 2024-03-25T14:32:43.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 >> v1:172.21.15.26:6790/0 conn(0x7f3dd006adf0 legacy=0x7f3dd013d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:43.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 >> v1:172.21.15.26:6789/0 conn(0x7f3dd00af580 legacy=0x7f3dd01402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:43.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: _finish_hunting 0 2024-03-25T14:32:43.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 1 monclient: found mon.noname-c 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd013dd40 con 0x7f3dd01409d0 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: _finish_auth 0 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: _check_auth_tickets 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3dc8001a10 con 0x7f3dd01409d0 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd7c86640 5 monclient: authenticate success, global_id 4465 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd7c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd4a20640 10 monclient: dump: 2024-03-25T14:32:43.982 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:43.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.975+0000 7f3dd7c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dc8001b70 con 0x7f3dd01409d0 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd4a20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd4a20640 1 -- 172.21.15.26:0/1443566688 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3dc8001f00 con 0x7f3dd01409d0 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd4a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd4a20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd4a20640 10 monclient: dump: 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:43.997 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd7c86640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:43.998 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.991+0000 7f3dd7c86640 4 set_mon_vals no callback set 2024-03-25T14:32:44.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 10 monclient: shutdown 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 1 -- 172.21.15.26:0/1443566688 >> v1:172.21.15.121:6789/0 conn(0x7f3dd01409d0 legacy=0x7f3dd014b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 1 -- 172.21.15.26:0/1443566688 shutdown_connections 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 1 -- 172.21.15.26:0/1443566688 >> 172.21.15.26:0/1443566688 conn(0x7f3dd00b4b80 msgr2=0x7f3dd00b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 1 -- 172.21.15.26:0/1443566688 shutdown_connections 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.992+0000 7f3dd7c86640 1 -- 172.21.15.26:0/1443566688 wait complete. 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.993+0000 7f3dd7c86640 10 monclient: build_initial_monmap 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.993+0000 7f3dd7c86640 10 monclient: monmap: 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:44.001 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.993+0000 7f3dd7c86640 1 librados: starting msgr at 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.993+0000 7f3dd7c86640 1 librados: starting objecter 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 Processor -- start 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 -- start start 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 librados: setting wanted keys 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 librados: calling monclient init 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient: init 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:32:44.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient(hunting): picked mon.noname-c con 0x7f3dd006adf0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient(hunting): picked mon.noname-b con 0x7f3dd00af580 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient(hunting): picked mon.noname-a con 0x7f3dd01409d0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd0135860 con 0x7f3dd01409d0 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd01340e0 con 0x7f3dd00af580 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd0135ad0 con 0x7f3dd006adf0 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:44.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:43.994+0000 7f3dd7c86640 10 monclient(hunting): authenticate will time out at 1849.098417s 2024-03-25T14:32:44.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.000+0000 7f3dd5a22640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3dd006adf0 0x7f3dd01339d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60350/0 (socket says 172.21.15.26:60350) 2024-03-25T14:32:44.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.000+0000 7f3dd5a22640 1 -- 172.21.15.26:0/3276104854 learned_addr learned my addr 172.21.15.26:0/3276104854 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:44.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.001+0000 7f3dc67fc640 10 client.?.objecter ms_handle_connect 0x7f3dd006adf0 2024-03-25T14:32:44.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.001+0000 7f3dc67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:44.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.006+0000 7f3dc67fc640 10 client.?.objecter ms_handle_connect 0x7f3dd00af580 2024-03-25T14:32:44.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.006+0000 7f3dc67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:44.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.012+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884805045 0 0) 0x7f3dd0135ad0 con 0x7f3dd006adf0 2024-03-25T14:32:44.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.012+0000 7f3dc67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:44.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.012+0000 7f3dc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:44.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.012+0000 7f3dc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:44.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.012+0000 7f3dc67fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T14:32:44.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.012+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db4004eb0 con 0x7f3dd006adf0 2024-03-25T14:32:44.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.015+0000 7f3dc67fc640 10 client.?.objecter ms_handle_connect 0x7f3dd01409d0 2024-03-25T14:32:44.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.015+0000 7f3dc67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:44.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.023+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518410857 0 0) 0x7f3dd01340e0 con 0x7f3dd00af580 2024-03-25T14:32:44.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.023+0000 7f3dc67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:44.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.023+0000 7f3dc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:44.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.023+0000 7f3dc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:44.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.023+0000 7f3dc67fc640 10 monclient(hunting): my global_id is 4468 2024-03-25T14:32:44.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.023+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd0135ad0 con 0x7f3dd00af580 2024-03-25T14:32:44.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.024+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071959991 0 0) 0x7f3dd0135860 con 0x7f3dd01409d0 2024-03-25T14:32:44.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.024+0000 7f3dc67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:32:44.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.024+0000 7f3dc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:44.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.024+0000 7f3dc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:44.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.024+0000 7f3dc67fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T14:32:44.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.024+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd01340e0 con 0x7f3dd01409d0 2024-03-25T14:32:44.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606037378 0 0) 0x7f3db4004eb0 con 0x7f3dd006adf0 2024-03-25T14:32:44.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dd0135860 con 0x7f3dd006adf0 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3dcc0019f0 con 0x7f3dd006adf0 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): dump: 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:44.039 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2550168850 0 0) 0x7f3dd01340e0 con 0x7f3dd01409d0 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3db4004eb0 con 0x7f3dd01409d0 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3dc8001ce0 con 0x7f3dd01409d0 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.032+0000 7f3dc67fc640 10 monclient(hunting): dump: 2024-03-25T14:32:44.040 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:44.041 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:44.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.035+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2466589544 0 0) 0x7f3dd0135ad0 con 0x7f3dd00af580 2024-03-25T14:32:44.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.035+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dd01340e0 con 0x7f3dd00af580 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.041+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3dc0001970 con 0x7f3dd00af580 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.041+0000 7f3dc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.041+0000 7f3dc67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.041+0000 7f3dc67fc640 10 monclient(hunting): dump: 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:44.047 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:44.048 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:44.048 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:44.048 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:44.048 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:44.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 948017389 0 0) 0x7f3dd0135860 con 0x7f3dd006adf0 2024-03-25T14:32:44.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 >> v1:172.21.15.121:6789/0 conn(0x7f3dd00af580 legacy=0x7f3dd0143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:44.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 >> v1:172.21.15.26:6789/0 conn(0x7f3dd01409d0 legacy=0x7f3dd01e0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:44.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 1 monclient: found mon.c 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd0135db0 con 0x7f3dd006adf0 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 10 monclient: _finish_auth 0 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 5 monclient: authenticate success, global_id 4439 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 10 monclient: _renew_subs 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd01d5d40 con 0x7f3dd006adf0 2024-03-25T14:32:44.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:44.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 10 monclient: _renew_subs 2024-03-25T14:32:44.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:44.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd0143d60 con 0x7f3dd006adf0 2024-03-25T14:32:44.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.043+0000 7f3dd7c86640 1 librados: init done 2024-03-25T14:32:44.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.044+0000 7f3dd7c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:44.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.044+0000 7f3dd7c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:44.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.044+0000 7f3dd7c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:44.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.044+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd00627b0 con 0x7f3dd006adf0 2024-03-25T14:32:44.364 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dcc0047e0 con 0x7f3dd006adf0 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3dcc004af0 con 0x7f3dd006adf0 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc67fc640 10 monclient: dump: 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:44.365 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:44.366 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:44.366 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:44.366 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:44.366 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.358+0000 7f3dc7fff640 4 set_mon_vals no callback set 2024-03-25T14:32:44.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.359+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f3dcc01ab40 con 0x7f3dd006adf0 2024-03-25T14:32:44.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.360+0000 7f3dd5221640 10 monclient: get_auth_request con 0x7f3db406e230 auth_method 0 2024-03-25T14:32:44.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.360+0000 7f3dc67fc640 10 client.4439.objecter ms_handle_connect 0x7f3db406e230 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f3dcc0548c0 con 0x7f3dd006adf0 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 10 client.4439.objecter ms_dispatch 0x7f3dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3dcc050200 con 0x7f3dd006adf0 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:44.375 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.367+0000 7f3dc67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:44.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.600+0000 7f3dd7c86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:44.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.600+0000 7f3dd7c86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:44.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.600+0000 7f3dd7c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:44.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.600+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dd00b0000 con 0x7f3dd006adf0 2024-03-25T14:32:44.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.621+0000 7f3dc67fc640 1 -- 172.21.15.26:0/3276104854 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f3dcc052300 con 0x7f3dd006adf0 2024-03-25T14:32:44.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.621+0000 7f3dc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:44.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.621+0000 7f3dc67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:44.627 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:44.627 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 10 librados: watch_flush enter 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 10 librados: watch_flush exit 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 10 client.4439.objecter successfully canceled tick 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 >> v1:172.21.15.121:6800/288813066 conn(0x7f3db406e230 legacy=0x7f3db40706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 10 monclient: shutdown 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 >> v1:172.21.15.26:6790/0 conn(0x7f3dd006adf0 legacy=0x7f3dd01339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:44.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.623+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 shutdown_connections 2024-03-25T14:32:44.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.624+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 >> 172.21.15.26:0/3276104854 conn(0x7f3dd00b4b80 msgr2=0x7f3dd0137380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:44.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.624+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 shutdown_connections 2024-03-25T14:32:44.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.624+0000 7f3dd7c86640 1 -- 172.21.15.26:0/3276104854 wait complete. 2024-03-25T14:32:44.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:44.624+0000 7f3dd7c86640 1 librados: shutdown 2024-03-25T14:32:44.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:51.453 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.564+0000 7fb2586a8640 10 monclient: get_monmap_and_config 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.564+0000 7fb2586a8640 10 monclient: build_initial_monmap 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.564+0000 7fb2586a8640 10 monclient: monmap: 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:51.565860+0000 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:51.565860+0000 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:51.570 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:51.571 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:51.571 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:51.571 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.564+0000 7fb2586a8640 1 Processor -- start 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.564+0000 7fb2586a8640 1 -- start start 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.564+0000 7fb2586a8640 10 monclient: init 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb2500a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb25013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb2501430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:51.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb250142520 con 0x7fb25013f710 2024-03-25T14:32:51.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2500b28a0 con 0x7fb2501430c0 2024-03-25T14:32:51.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2500b2a30 con 0x7fb2500a98f0 2024-03-25T14:32:51.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:51.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.565+0000 7fb2586a8640 10 monclient(hunting): authenticate will time out at 1856.669269s 2024-03-25T14:32:51.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.576+0000 7fb255c43640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb25013f710 0x7fb2501429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37016/0 (socket says 172.21.15.26:37016) 2024-03-25T14:32:51.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.576+0000 7fb255c43640 1 -- 172.21.15.26:0/2060239106 learned_addr learned my addr 172.21.15.26:0/2060239106 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:51.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.589+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977783567 0 0) 0x7fb2500b28a0 con 0x7fb2501430c0 2024-03-25T14:32:51.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.589+0000 7fb255442640 10 monclient(hunting): _negotiate 2024-03-25T14:32:51.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.589+0000 7fb255442640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:51.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.589+0000 7fb255442640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:51.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.589+0000 7fb255442640 10 monclient(hunting): my global_id is 4471 2024-03-25T14:32:51.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.589+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb23c003520 con 0x7fb2501430c0 2024-03-25T14:32:51.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.606+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673443511 0 0) 0x7fb250142520 con 0x7fb25013f710 2024-03-25T14:32:51.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.606+0000 7fb255442640 10 monclient(hunting): _negotiate 2024-03-25T14:32:51.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.606+0000 7fb255442640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:51.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.606+0000 7fb255442640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:51.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.606+0000 7fb255442640 10 monclient(hunting): my global_id is 4485 2024-03-25T14:32:51.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.606+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2500b28a0 con 0x7fb25013f710 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346077639 0 0) 0x7fb2500b2a30 con 0x7fb2500a98f0 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient(hunting): _negotiate 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient(hunting): my global_id is 4442 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250142520 con 0x7fb2500a98f0 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536071675 0 0) 0x7fb23c003520 con 0x7fb2501430c0 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 >> v1:172.21.15.26:6790/0 conn(0x7fb2500a98f0 legacy=0x7fb2500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 >> v1:172.21.15.26:6789/0 conn(0x7fb25013f710 legacy=0x7fb2501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: _finish_hunting 0 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 monclient: found mon.noname-c 2024-03-25T14:32:51.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2500b44c0 con 0x7fb2501430c0 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: _finish_auth 0 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: _check_auth_tickets 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb248001a10 con 0x7fb2501430c0 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb2586a8640 5 monclient: authenticate success, global_id 4471 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb2586a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb255442640 10 monclient: dump: 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:51.617 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.609+0000 7fb2586a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb248001b70 con 0x7fb2501430c0 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb255442640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb255442640 1 -- 172.21.15.26:0/2060239106 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb248001f80 con 0x7fb2501430c0 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb2586a8640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb2586a8640 4 set_mon_vals no callback set 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb255442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb255442640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:51.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.640+0000 7fb255442640 10 monclient: dump: 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:51.647 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 10 monclient: shutdown 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 1 -- 172.21.15.26:0/2060239106 >> v1:172.21.15.121:6789/0 conn(0x7fb2501430c0 legacy=0x7fb25014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 1 -- 172.21.15.26:0/2060239106 shutdown_connections 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 1 -- 172.21.15.26:0/2060239106 >> 172.21.15.26:0/2060239106 conn(0x7fb2500b7ea0 msgr2=0x7fb2500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 1 -- 172.21.15.26:0/2060239106 shutdown_connections 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.641+0000 7fb2586a8640 1 -- 172.21.15.26:0/2060239106 wait complete. 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 10 monclient: build_initial_monmap 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 10 monclient: monmap: 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:51.649 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 1 librados: starting msgr at 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 1 librados: starting objecter 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 1 Processor -- start 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 1 -- start start 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 1 librados: setting wanted keys 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 1 librados: calling monclient init 2024-03-25T14:32:51.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.642+0000 7fb2586a8640 10 monclient: init 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb2500a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb25013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb2501430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb250137c60 con 0x7fb2500a98f0 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb250139160 con 0x7fb2501430c0 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2501380f0 con 0x7fb25013f710 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:51.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.643+0000 7fb2586a8640 10 monclient(hunting): authenticate will time out at 1856.747591s 2024-03-25T14:32:51.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.656+0000 7fb255c43640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb25013f710 0x7fb250133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59094/0 (socket says 172.21.15.26:59094) 2024-03-25T14:32:51.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.656+0000 7fb255c43640 1 -- 172.21.15.26:0/2622337912 learned_addr learned my addr 172.21.15.26:0/2622337912 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:51.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.657+0000 7fb246ffd640 10 client.?.objecter ms_handle_connect 0x7fb25013f710 2024-03-25T14:32:51.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.657+0000 7fb246ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:51.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.663+0000 7fb246ffd640 10 client.?.objecter ms_handle_connect 0x7fb2501430c0 2024-03-25T14:32:51.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.663+0000 7fb246ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:51.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.664+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528268712 0 0) 0x7fb2501380f0 con 0x7fb25013f710 2024-03-25T14:32:51.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.664+0000 7fb246ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:51.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.664+0000 7fb246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:51.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.664+0000 7fb246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:51.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.664+0000 7fb246ffd640 10 monclient(hunting): my global_id is 4445 2024-03-25T14:32:51.671 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.664+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234004eb0 con 0x7fb25013f710 2024-03-25T14:32:51.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.673+0000 7fb246ffd640 10 client.?.objecter ms_handle_connect 0x7fb2500a98f0 2024-03-25T14:32:51.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.673+0000 7fb246ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677232785 0 0) 0x7fb250139160 con 0x7fb2501430c0 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): my global_id is 4474 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2501380f0 con 0x7fb2501430c0 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205869824 0 0) 0x7fb250137c60 con 0x7fb2500a98f0 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:51.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T14:32:51.686 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.679+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250139160 con 0x7fb2500a98f0 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531780159 0 0) 0x7fb234004eb0 con 0x7fb25013f710 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb250137c60 con 0x7fb25013f710 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb2400019f0 con 0x7fb25013f710 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 10 monclient(hunting): dump: 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:51.690 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:51.691 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:51.691 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:51.691 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:51.691 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.684+0000 7fb246ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:51.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.699+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1640344696 0 0) 0x7fb250139160 con 0x7fb2500a98f0 2024-03-25T14:32:51.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.699+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb234004eb0 con 0x7fb2500a98f0 2024-03-25T14:32:51.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.699+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb24c001830 con 0x7fb2500a98f0 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.699+0000 7fb246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.699+0000 7fb246ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.699+0000 7fb246ffd640 10 monclient(hunting): dump: 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:51.706 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:51.707 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:51.707 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:51.707 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.700+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 527053797 0 0) 0x7fb2501380f0 con 0x7fb2501430c0 2024-03-25T14:32:51.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.700+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb250139160 con 0x7fb2501430c0 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.700+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb248001e80 con 0x7fb2501430c0 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.700+0000 7fb246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.700+0000 7fb246ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.700+0000 7fb246ffd640 10 monclient(hunting): dump: 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:51.708 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:51.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 27514208 0 0) 0x7fb250137c60 con 0x7fb25013f710 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 >> v1:172.21.15.121:6789/0 conn(0x7fb2501430c0 legacy=0x7fb2501e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 >> v1:172.21.15.26:6789/0 conn(0x7fb2500a98f0 legacy=0x7fb2501dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 1 monclient: found mon.c 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2501397f0 con 0x7fb25013f710 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 10 monclient: _finish_auth 0 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 5 monclient: authenticate success, global_id 4445 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 10 monclient: _renew_subs 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:51.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2501db280 con 0x7fb25013f710 2024-03-25T14:32:51.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:51.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 10 monclient: _renew_subs 2024-03-25T14:32:51.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:51.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb250133780 con 0x7fb25013f710 2024-03-25T14:32:51.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.706+0000 7fb2586a8640 1 librados: init done 2024-03-25T14:32:51.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.707+0000 7fb2586a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:51.715 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.707+0000 7fb2586a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:51.715 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.707+0000 7fb2586a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:51.715 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:51.707+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb250139a40 con 0x7fb25013f710 2024-03-25T14:32:52.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2400047e0 con 0x7fb25013f710 2024-03-25T14:32:52.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb246ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fb240004af0 con 0x7fb25013f710 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb246ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb246ffd640 10 monclient: dump: 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:52.023 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:52.024 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.016+0000 7fb247fff640 4 set_mon_vals no callback set 2024-03-25T14:32:52.024 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.017+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fb24001ab80 con 0x7fb25013f710 2024-03-25T14:32:52.024 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.018+0000 7fb256444640 10 monclient: get_auth_request con 0x7fb23406e0f0 auth_method 0 2024-03-25T14:32:52.024 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.018+0000 7fb246ffd640 10 client.4445.objecter ms_handle_connect 0x7fb23406e0f0 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fb240054860 con 0x7fb25013f710 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 10 client.4445.objecter ms_dispatch 0x7fb250000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2400501a0 con 0x7fb25013f710 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:52.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.023+0000 7fb246ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:52.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.265+0000 7fb2586a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:52.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.265+0000 7fb2586a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:52.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.265+0000 7fb2586a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:32:52.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.265+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb250137e10 con 0x7fb25013f710 2024-03-25T14:32:52.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.287+0000 7fb246ffd640 1 -- 172.21.15.26:0/2622337912 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fb250137e10 con 0x7fb25013f710 2024-03-25T14:32:52.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.287+0000 7fb246ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:52.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.287+0000 7fb246ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:52.294 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:52.294 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:52.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 10 librados: watch_flush enter 2024-03-25T14:32:52.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 10 librados: watch_flush exit 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 10 client.4445.objecter successfully canceled tick 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 >> v1:172.21.15.121:6800/288813066 conn(0x7fb23406e0f0 legacy=0x7fb2340705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 10 monclient: shutdown 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 >> v1:172.21.15.26:6790/0 conn(0x7fb25013f710 legacy=0x7fb250133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 shutdown_connections 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 >> 172.21.15.26:0/2622337912 conn(0x7fb2500b7ea0 msgr2=0x7fb25013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 shutdown_connections 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 -- 172.21.15.26:0/2622337912 wait complete. 2024-03-25T14:32:52.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:52.290+0000 7fb2586a8640 1 librados: shutdown 2024-03-25T14:32:52.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:32:56.803 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:06:15.663 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.815 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:06:15.675 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.857 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:06:15.717 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.867 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:06:15.743 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.872 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:06:15.748 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.887 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:06:15.747 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.890 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:06:15.751 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.897 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:06:15.773 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:56.933 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:06:15.793 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:32:59.115 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:32:59.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.226+0000 7f6a28465640 10 monclient: get_monmap_and_config 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.226+0000 7f6a28465640 10 monclient: build_initial_monmap 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.226+0000 7f6a28465640 10 monclient: monmap: 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:32:59.227758+0000 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:32:59.227758+0000 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:32:59.232 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:32:59.233 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:59.234 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.227+0000 7f6a28465640 1 Processor -- start 2024-03-25T14:32:59.234 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 1 -- start start 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 10 monclient: init 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 10 monclient(hunting): picked mon.noname-c con 0x7f6a200b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 10 monclient(hunting): picked mon.noname-a con 0x7f6a200b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 10 monclient(hunting): picked mon.noname-b con 0x7f6a2014be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a200af940 con 0x7f6a200b1730 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.228+0000 7f6a28465640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a200b0a70 con 0x7f6a200b4d60 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.229+0000 7f6a28465640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a200b0d50 con 0x7f6a2014be90 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.229+0000 7f6a28465640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:59.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.229+0000 7f6a28465640 10 monclient(hunting): authenticate will time out at 1864.331772s 2024-03-25T14:32:59.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.238+0000 7f6a26201640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6a200b4d60 0x7f6a2013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39374/0 (socket says 172.21.15.26:39374) 2024-03-25T14:32:59.243 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.238+0000 7f6a26201640 1 -- 172.21.15.26:0/261882800 learned_addr learned my addr 172.21.15.26:0/261882800 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:59.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.250+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893728732 0 0) 0x7f6a200b0a70 con 0x7f6a200b4d60 2024-03-25T14:32:59.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.250+0000 7f6a251ff640 10 monclient(hunting): _negotiate 2024-03-25T14:32:59.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.250+0000 7f6a251ff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:59.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.250+0000 7f6a251ff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:59.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.250+0000 7f6a251ff640 10 monclient(hunting): my global_id is 4477 2024-03-25T14:32:59.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.250+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c003520 con 0x7f6a200b4d60 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1146642943 0 0) 0x7f6a0c003520 con 0x7f6a200b4d60 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 >> v1:172.21.15.26:6790/0 conn(0x7f6a2014be90 legacy=0x7f6a2014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 >> v1:172.21.15.26:6789/0 conn(0x7f6a200b1730 legacy=0x7f6a200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: _finish_hunting 0 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 1 monclient: found mon.noname-c 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200b1030 con 0x7f6a200b4d60 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: _finish_auth 0 2024-03-25T14:32:59.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: _check_auth_tickets 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6a1c001690 con 0x7f6a200b4d60 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a28465640 5 monclient: authenticate success, global_id 4477 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a28465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a251ff640 10 monclient: dump: 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.272 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:59.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.265+0000 7f6a28465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a1c0017f0 con 0x7f6a200b4d60 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a251ff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a251ff640 1 -- 172.21.15.26:0/261882800 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6a1c001c00 con 0x7f6a200b4d60 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a251ff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a251ff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a251ff640 10 monclient: dump: 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:59.281 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:59.282 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:59.282 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:59.282 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:59.282 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:59.282 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a28465640 10 monclient: get_monmap_and_config success 2024-03-25T14:32:59.282 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a28465640 4 set_mon_vals no callback set 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a28465640 10 monclient: shutdown 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a28465640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.276+0000 7f6a28465640 1 -- 172.21.15.26:0/261882800 >> v1:172.21.15.121:6789/0 conn(0x7f6a200b4d60 legacy=0x7f6a2013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.277+0000 7f6a28465640 1 -- 172.21.15.26:0/261882800 shutdown_connections 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.277+0000 7f6a28465640 1 -- 172.21.15.26:0/261882800 >> 172.21.15.26:0/261882800 conn(0x7f6a200617a0 msgr2=0x7f6a20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.277+0000 7f6a28465640 1 -- 172.21.15.26:0/261882800 shutdown_connections 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.277+0000 7f6a28465640 1 -- 172.21.15.26:0/261882800 wait complete. 2024-03-25T14:32:59.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 10 monclient: build_initial_monmap 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 10 monclient: monmap: 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 1 librados: starting msgr at 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 1 librados: starting objecter 2024-03-25T14:32:59.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 1 Processor -- start 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 1 -- start start 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 1 librados: setting wanted keys 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 1 librados: calling monclient init 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.278+0000 7f6a28465640 10 monclient: init 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient: _reopen_session rank -1 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient(hunting): picked mon.noname-a con 0x7f6a200b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient(hunting): picked mon.noname-b con 0x7f6a200b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2014be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a20142050 con 0x7f6a200b1730 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a201dac50 con 0x7f6a200b4d60 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a20140d00 con 0x7f6a2014be90 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient(hunting): _renew_subs 2024-03-25T14:32:59.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.279+0000 7f6a28465640 10 monclient(hunting): authenticate will time out at 1864.381841s 2024-03-25T14:32:59.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.292+0000 7f6a25a00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6a200b4d60 0x7f6a201db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39408/0 (socket says 172.21.15.26:39408) 2024-03-25T14:32:59.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.292+0000 7f6a25a00640 1 -- 172.21.15.26:0/1392574992 learned_addr learned my addr 172.21.15.26:0/1392574992 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:32:59.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.296+0000 7f6a16ffd640 10 client.?.objecter ms_handle_connect 0x7f6a200b4d60 2024-03-25T14:32:59.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.296+0000 7f6a16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:59.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.300+0000 7f6a16ffd640 10 client.?.objecter ms_handle_connect 0x7f6a2014be90 2024-03-25T14:32:59.305 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.300+0000 7f6a16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:59.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.305+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987426226 0 0) 0x7f6a201dac50 con 0x7f6a200b4d60 2024-03-25T14:32:59.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.305+0000 7f6a16ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:59.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.305+0000 7f6a16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:59.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.305+0000 7f6a16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:59.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.305+0000 7f6a16ffd640 10 monclient(hunting): my global_id is 4480 2024-03-25T14:32:59.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.305+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a04004eb0 con 0x7f6a200b4d60 2024-03-25T14:32:59.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.310+0000 7f6a16ffd640 10 client.?.objecter ms_handle_connect 0x7f6a200b1730 2024-03-25T14:32:59.315 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.310+0000 7f6a16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:32:59.326 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.321+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231000315 0 0) 0x7f6a20140d00 con 0x7f6a2014be90 2024-03-25T14:32:59.326 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.321+0000 7f6a16ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:59.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.321+0000 7f6a16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:59.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.321+0000 7f6a16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:59.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.321+0000 7f6a16ffd640 10 monclient(hunting): my global_id is 4448 2024-03-25T14:32:59.327 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.321+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a201dac50 con 0x7f6a2014be90 2024-03-25T14:32:59.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.327+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765235612 0 0) 0x7f6a20142050 con 0x7f6a200b1730 2024-03-25T14:32:59.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.327+0000 7f6a16ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:32:59.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.327+0000 7f6a16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:32:59.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.327+0000 7f6a16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:32:59.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.327+0000 7f6a16ffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T14:32:59.332 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.327+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a20140d00 con 0x7f6a200b1730 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1850062081 0 0) 0x7f6a04004eb0 con 0x7f6a200b4d60 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a20142050 con 0x7f6a200b4d60 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6a100019d0 con 0x7f6a200b4d60 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 10 monclient(hunting): dump: 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.335 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:59.336 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.330+0000 7f6a16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:32:59.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2547500814 0 0) 0x7f6a20142050 con 0x7f6a200b4d60 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 >> v1:172.21.15.26:6790/0 conn(0x7f6a2014be90 legacy=0x7f6a201e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 >> v1:172.21.15.26:6789/0 conn(0x7f6a200b1730 legacy=0x7f6a201405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 1 monclient: found mon.b 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a201423e0 con 0x7f6a200b4d60 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 10 monclient: _finish_auth 0 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 5 monclient: authenticate success, global_id 4480 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 10 monclient: _renew_subs 2024-03-25T14:32:59.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2013b470 con 0x7f6a200b4d60 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 10 monclient: _renew_subs 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a201dba40 con 0x7f6a200b4d60 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.338+0000 7f6a28465640 1 librados: init done 2024-03-25T14:32:59.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.339+0000 7f6a28465640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:59.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.339+0000 7f6a28465640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:59.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.339+0000 7f6a28465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:59.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.339+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a201401f0 con 0x7f6a200b4d60 2024-03-25T14:32:59.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a100035f0 con 0x7f6a200b4d60 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f6a10003a00 con 0x7f6a200b4d60 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a16ffd640 10 monclient: dump: 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:32:59.619 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:32:59.620 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:32:59.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.613+0000 7f6a249fe640 4 set_mon_vals no callback set 2024-03-25T14:32:59.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.614+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f6a10003c20 con 0x7f6a200b4d60 2024-03-25T14:32:59.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.615+0000 7f6a26201640 10 monclient: get_auth_request con 0x7f6a0406de20 auth_method 0 2024-03-25T14:32:59.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.615+0000 7f6a16ffd640 10 client.4480.objecter ms_handle_connect 0x7f6a0406de20 2024-03-25T14:32:59.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.618+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f6a10054820 con 0x7f6a200b4d60 2024-03-25T14:32:59.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.618+0000 7f6a16ffd640 10 client.4480.objecter ms_dispatch 0x7f6a20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:32:59.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.618+0000 7f6a16ffd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:32:59.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.618+0000 7f6a16ffd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:32:59.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.619+0000 7f6a16ffd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T14:32:59.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.619+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a10050330 con 0x7f6a200b4d60 2024-03-25T14:32:59.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.619+0000 7f6a16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:32:59.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.619+0000 7f6a16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:32:59.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.878+0000 7f6a28465640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:59.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.878+0000 7f6a28465640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:59.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.878+0000 7f6a28465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:32:59.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.878+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a20140e90 con 0x7f6a200b4d60 2024-03-25T14:32:59.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.904+0000 7f6a16ffd640 1 -- 172.21.15.26:0/1392574992 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f6a10052260 con 0x7f6a200b4d60 2024-03-25T14:32:59.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.904+0000 7f6a16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:32:59.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.904+0000 7f6a16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:32:59.909 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:32:59.909 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:32:59.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 10 librados: watch_flush enter 2024-03-25T14:32:59.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 10 librados: watch_flush exit 2024-03-25T14:32:59.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 10 client.4480.objecter successfully canceled tick 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 >> v1:172.21.15.121:6800/288813066 conn(0x7f6a0406de20 legacy=0x7f6a040702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 10 monclient: shutdown 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 >> v1:172.21.15.121:6789/0 conn(0x7f6a200b4d60 legacy=0x7f6a201db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 shutdown_connections 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 >> 172.21.15.26:0/1392574992 conn(0x7f6a200617a0 msgr2=0x7f6a200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 shutdown_connections 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 -- 172.21.15.26:0/1392574992 wait complete. 2024-03-25T14:32:59.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:32:59.906+0000 7f6a28465640 1 librados: shutdown 2024-03-25T14:32:59.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:06.731 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:06.840 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.835+0000 7fa959e7b640 10 monclient: get_monmap_and_config 2024-03-25T14:33:06.840 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.835+0000 7fa959e7b640 10 monclient: build_initial_monmap 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.835+0000 7fa959e7b640 10 monclient: monmap: 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:06.836503+0000 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:06.836503+0000 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:06.841 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.836+0000 7fa959e7b640 1 Processor -- start 2024-03-25T14:33:06.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.836+0000 7fa959e7b640 1 -- start start 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.836+0000 7fa959e7b640 10 monclient: init 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa9540a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa95413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa9541430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa954142520 con 0x7fa9541430c0 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9540b28a0 con 0x7fa95413f710 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9540b2a30 con 0x7fa9540a98f0 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:06.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.837+0000 7fa959e7b640 10 monclient(hunting): authenticate will time out at 1871.940078s 2024-03-25T14:33:06.856 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.852+0000 7fa952ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa95413f710 0x7fa9541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:44324/0 (socket says 172.21.15.26:44324) 2024-03-25T14:33:06.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.852+0000 7fa952ffd640 1 -- 172.21.15.26:0/4057062995 learned_addr learned my addr 172.21.15.26:0/4057062995 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:06.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.867+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809498114 0 0) 0x7fa954142520 con 0x7fa9541430c0 2024-03-25T14:33:06.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.867+0000 7fa9527fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:06.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.867+0000 7fa9527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:06.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.867+0000 7fa9527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:06.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.867+0000 7fa9527fc640 10 monclient(hunting): my global_id is 4497 2024-03-25T14:33:06.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.867+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93c003520 con 0x7fa9541430c0 2024-03-25T14:33:06.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.873+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300212524 0 0) 0x7fa9540b28a0 con 0x7fa95413f710 2024-03-25T14:33:06.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.873+0000 7fa9527fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:06.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.873+0000 7fa9527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:06.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.873+0000 7fa9527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:06.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.873+0000 7fa9527fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T14:33:06.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.873+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa954142520 con 0x7fa95413f710 2024-03-25T14:33:06.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1805018063 0 0) 0x7fa93c003520 con 0x7fa9541430c0 2024-03-25T14:33:06.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 >> v1:172.21.15.26:6790/0 conn(0x7fa9540a98f0 legacy=0x7fa9540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:06.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 >> v1:172.21.15.121:6789/0 conn(0x7fa95413f710 legacy=0x7fa9541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:06.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: _finish_hunting 0 2024-03-25T14:33:06.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 1 monclient: found mon.noname-a 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540b44c0 con 0x7fa9541430c0 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: _finish_auth 0 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: _check_auth_tickets 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa94c001a10 con 0x7fa9541430c0 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa9527fc640 10 monclient: dump: 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:06.883 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa959e7b640 5 monclient: authenticate success, global_id 4497 2024-03-25T14:33:06.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.877+0000 7fa959e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa94c001b70 con 0x7fa9541430c0 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa9527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa9527fc640 1 -- 172.21.15.26:0/4057062995 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa94c001f80 con 0x7fa9541430c0 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa959e7b640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa959e7b640 4 set_mon_vals no callback set 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa9527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa9527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.891+0000 7fa9527fc640 10 monclient: dump: 2024-03-25T14:33:06.896 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:06.897 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:06.898 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 10 monclient: shutdown 2024-03-25T14:33:06.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:06.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 1 -- 172.21.15.26:0/4057062995 >> v1:172.21.15.26:6789/0 conn(0x7fa9541430c0 legacy=0x7fa95414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:06.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 1 -- 172.21.15.26:0/4057062995 shutdown_connections 2024-03-25T14:33:06.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 1 -- 172.21.15.26:0/4057062995 >> 172.21.15.26:0/4057062995 conn(0x7fa9540b7ea0 msgr2=0x7fa9540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 1 -- 172.21.15.26:0/4057062995 shutdown_connections 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.892+0000 7fa959e7b640 1 -- 172.21.15.26:0/4057062995 wait complete. 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 10 monclient: build_initial_monmap 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 10 monclient: monmap: 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:06.901 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 1 librados: starting msgr at 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 1 librados: starting objecter 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 1 Processor -- start 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 1 -- start start 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 1 librados: setting wanted keys 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 1 librados: calling monclient init 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.893+0000 7fa959e7b640 10 monclient: init 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa9540a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa95413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:06.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa9541430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:06.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa954139830 con 0x7fa95413f710 2024-03-25T14:33:06.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9541db7d0 con 0x7fa9541430c0 2024-03-25T14:33:06.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9541db410 con 0x7fa9540a98f0 2024-03-25T14:33:06.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:06.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.894+0000 7fa959e7b640 10 monclient(hunting): authenticate will time out at 1871.996950s 2024-03-25T14:33:06.907 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.902+0000 7fa9537fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa9540a98f0 0x7fa9541dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:56908/0 (socket says 172.21.15.26:56908) 2024-03-25T14:33:06.907 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.902+0000 7fa9537fe640 1 -- 172.21.15.26:0/982857961 learned_addr learned my addr 172.21.15.26:0/982857961 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:06.907 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.903+0000 7fa93bfff640 10 client.?.objecter ms_handle_connect 0x7fa9540a98f0 2024-03-25T14:33:06.907 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.903+0000 7fa93bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:06.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.907+0000 7fa93bfff640 10 client.?.objecter ms_handle_connect 0x7fa9541430c0 2024-03-25T14:33:06.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.907+0000 7fa93bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:06.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.909+0000 7fa93bfff640 10 client.?.objecter ms_handle_connect 0x7fa95413f710 2024-03-25T14:33:06.914 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.909+0000 7fa93bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.910+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680537988 0 0) 0x7fa9541db410 con 0x7fa9540a98f0 2024-03-25T14:33:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.910+0000 7fa93bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.910+0000 7fa93bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.910+0000 7fa93bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.910+0000 7fa93bfff640 10 monclient(hunting): my global_id is 4454 2024-03-25T14:33:06.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.910+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa940004eb0 con 0x7fa9540a98f0 2024-03-25T14:33:06.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.921+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787387749 0 0) 0x7fa9541db7d0 con 0x7fa9541430c0 2024-03-25T14:33:06.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.921+0000 7fa93bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:06.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.921+0000 7fa93bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:06.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.921+0000 7fa93bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:06.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.921+0000 7fa93bfff640 10 monclient(hunting): my global_id is 4486 2024-03-25T14:33:06.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.921+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9541db410 con 0x7fa9541430c0 2024-03-25T14:33:06.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.922+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596426040 0 0) 0x7fa954139830 con 0x7fa95413f710 2024-03-25T14:33:06.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.922+0000 7fa93bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:06.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.922+0000 7fa93bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:06.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.922+0000 7fa93bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:06.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.922+0000 7fa93bfff640 10 monclient(hunting): my global_id is 4500 2024-03-25T14:33:06.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.922+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9541db7d0 con 0x7fa95413f710 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 948699288 0 0) 0x7fa940004eb0 con 0x7fa9540a98f0 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa954139830 con 0x7fa9540a98f0 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa944001a30 con 0x7fa9540a98f0 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 10 monclient(hunting): dump: 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:06.930 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.925+0000 7fa93bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.932+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1424207317 0 0) 0x7fa9541db410 con 0x7fa9541430c0 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.933+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa940004eb0 con 0x7fa9541430c0 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.933+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa94c001e80 con 0x7fa9541430c0 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.933+0000 7fa93bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.933+0000 7fa93bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.933+0000 7fa93bfff640 10 monclient(hunting): dump: 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.938 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:06.939 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:06.939 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:06.939 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:06.939 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:06.939 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.939+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3672915228 0 0) 0x7fa9541db7d0 con 0x7fa95413f710 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.939+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9541db410 con 0x7fa95413f710 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.939+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa948001930 con 0x7fa95413f710 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.939+0000 7fa93bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.939+0000 7fa93bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.939+0000 7fa93bfff640 10 monclient(hunting): dump: 2024-03-25T14:33:06.944 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:06.945 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 534835948 0 0) 0x7fa9541db410 con 0x7fa95413f710 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 >> v1:172.21.15.26:6790/0 conn(0x7fa9540a98f0 legacy=0x7fa9541dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 >> v1:172.21.15.121:6789/0 conn(0x7fa9541430c0 legacy=0x7fa9541d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 1 monclient: found mon.a 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9541db5a0 con 0x7fa95413f710 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 10 monclient: _finish_auth 0 2024-03-25T14:33:06.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa93bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 5 monclient: authenticate success, global_id 4500 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 10 monclient: _renew_subs 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9540b1480 con 0x7fa95413f710 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 10 monclient: _renew_subs 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9540b0b10 con 0x7fa95413f710 2024-03-25T14:33:06.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.945+0000 7fa959e7b640 1 librados: init done 2024-03-25T14:33:06.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.946+0000 7fa959e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:06.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.946+0000 7fa959e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:06.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.946+0000 7fa959e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:33:06.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:06.946+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9540627b0 con 0x7fa95413f710 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.293+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9480047e0 con 0x7fa95413f710 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.293+0000 7fa93bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.293+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa948001d80 con 0x7fa95413f710 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.293+0000 7fa93bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.294+0000 7fa93bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.294+0000 7fa93bfff640 10 monclient: dump: 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:07.299 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:07.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.294+0000 7fa951ffb640 4 set_mon_vals no callback set 2024-03-25T14:33:07.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.294+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fa94801a9e0 con 0x7fa95413f710 2024-03-25T14:33:07.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.295+0000 7fa9537fe640 10 monclient: get_auth_request con 0x7fa94006e110 auth_method 0 2024-03-25T14:33:07.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.295+0000 7fa93bfff640 10 client.4500.objecter ms_handle_connect 0x7fa94006e110 2024-03-25T14:33:07.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fa948054730 con 0x7fa95413f710 2024-03-25T14:33:07.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 10 client.4500.objecter ms_dispatch 0x7fa954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:07.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:07.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:07.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-25T14:33:07.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa948050070 con 0x7fa95413f710 2024-03-25T14:33:07.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:07.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.301+0000 7fa93bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:07.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.554+0000 7fa959e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:07.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.554+0000 7fa959e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:07.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.554+0000 7fa959e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:33:07.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.554+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa954069f40 con 0x7fa95413f710 2024-03-25T14:33:07.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.599+0000 7fa93bfff640 1 -- 172.21.15.26:0/982857961 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fa948052170 con 0x7fa95413f710 2024-03-25T14:33:07.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.599+0000 7fa93bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:07.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.599+0000 7fa93bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:07.605 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:07.605 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:07.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 10 librados: watch_flush enter 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 10 librados: watch_flush exit 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 10 client.4500.objecter successfully canceled tick 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 >> v1:172.21.15.121:6800/288813066 conn(0x7fa94006e110 legacy=0x7fa9400705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 10 monclient: shutdown 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 >> v1:172.21.15.26:6789/0 conn(0x7fa95413f710 legacy=0x7fa9540b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 shutdown_connections 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 >> 172.21.15.26:0/982857961 conn(0x7fa9540b7ea0 msgr2=0x7fa954074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 shutdown_connections 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 -- 172.21.15.26:0/982857961 wait complete. 2024-03-25T14:33:07.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:07.602+0000 7fa959e7b640 1 librados: shutdown 2024-03-25T14:33:07.618 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:14.425 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:14.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.540+0000 7f5b2bc63640 10 monclient: get_monmap_and_config 2024-03-25T14:33:14.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.540+0000 7f5b2bc63640 10 monclient: build_initial_monmap 2024-03-25T14:33:14.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.540+0000 7f5b2bc63640 10 monclient: monmap: 2024-03-25T14:33:14.545 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:14.541156+0000 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:14.541156+0000 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:14.546 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.541+0000 7f5b2bc63640 1 Processor -- start 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.541+0000 7f5b2bc63640 1 -- start start 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.541+0000 7f5b2bc63640 10 monclient: init 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient(hunting): picked mon.noname-b con 0x7f5b240a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient(hunting): picked mon.noname-c con 0x7f5b241430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24142520 con 0x7f5b2413f710 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b28a0 con 0x7f5b241430c0 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b2a30 con 0x7f5b240a98f0 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:14.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.542+0000 7f5b2bc63640 10 monclient(hunting): authenticate will time out at 1879.645276s 2024-03-25T14:33:14.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.558+0000 7f5b299ff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5b240a98f0 0x7f5b240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40478/0 (socket says 172.21.15.26:40478) 2024-03-25T14:33:14.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.558+0000 7f5b299ff640 1 -- 172.21.15.26:0/1826006561 learned_addr learned my addr 172.21.15.26:0/1826006561 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:14.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.577+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289723945 0 0) 0x7f5b240b2a30 con 0x7f5b240a98f0 2024-03-25T14:33:14.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.577+0000 7f5b289fd640 10 monclient(hunting): _negotiate 2024-03-25T14:33:14.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.577+0000 7f5b289fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:14.583 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.577+0000 7f5b289fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:14.583 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.577+0000 7f5b289fd640 10 monclient(hunting): my global_id is 4457 2024-03-25T14:33:14.583 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.577+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c003520 con 0x7f5b240a98f0 2024-03-25T14:33:14.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.589+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696905647 0 0) 0x7f5b240b28a0 con 0x7f5b241430c0 2024-03-25T14:33:14.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.589+0000 7f5b289fd640 10 monclient(hunting): _negotiate 2024-03-25T14:33:14.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.589+0000 7f5b289fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:14.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.589+0000 7f5b289fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:14.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.589+0000 7f5b289fd640 10 monclient(hunting): my global_id is 4489 2024-03-25T14:33:14.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.589+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240b2a30 con 0x7f5b241430c0 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159454192 0 0) 0x7f5b240b2a30 con 0x7f5b241430c0 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 >> v1:172.21.15.26:6790/0 conn(0x7f5b240a98f0 legacy=0x7f5b240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 >> v1:172.21.15.26:6789/0 conn(0x7f5b2413f710 legacy=0x7f5b241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 10 monclient: _finish_hunting 0 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 1 monclient: found mon.noname-c 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240b44c0 con 0x7f5b241430c0 2024-03-25T14:33:14.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 10 monclient: _finish_auth 0 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 10 monclient: _check_auth_tickets 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5b1c0019f0 con 0x7f5b241430c0 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.603+0000 7f5b289fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.604+0000 7f5b289fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.604+0000 7f5b289fd640 10 monclient: dump: 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:14.610 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:14.611 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:14.611 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:14.611 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.604+0000 7f5b2bc63640 5 monclient: authenticate success, global_id 4489 2024-03-25T14:33:14.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.604+0000 7f5b2bc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:14.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b1c001b50 con 0x7f5b241430c0 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b289fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b289fd640 1 -- 172.21.15.26:0/1826006561 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5b1c001f20 con 0x7f5b241430c0 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b289fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b289fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b289fd640 10 monclient: dump: 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:14.637 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:14.638 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:14.638 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b2bc63640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:14.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.631+0000 7f5b2bc63640 4 set_mon_vals no callback set 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 10 monclient: shutdown 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 1 -- 172.21.15.26:0/1826006561 >> v1:172.21.15.121:6789/0 conn(0x7f5b241430c0 legacy=0x7f5b2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 1 -- 172.21.15.26:0/1826006561 shutdown_connections 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 1 -- 172.21.15.26:0/1826006561 >> 172.21.15.26:0/1826006561 conn(0x7f5b240b7ea0 msgr2=0x7f5b240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 1 -- 172.21.15.26:0/1826006561 shutdown_connections 2024-03-25T14:33:14.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.632+0000 7f5b2bc63640 1 -- 172.21.15.26:0/1826006561 wait complete. 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 10 monclient: build_initial_monmap 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 10 monclient: monmap: 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 1 librados: starting msgr at 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 1 librados: starting objecter 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 1 Processor -- start 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 1 -- start start 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 1 librados: setting wanted keys 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 1 librados: calling monclient init 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.633+0000 7f5b2bc63640 10 monclient: init 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient(hunting): picked mon.noname-c con 0x7f5b240a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient(hunting): picked mon.noname-b con 0x7f5b241430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:14.642 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b02b0 con 0x7f5b2413f710 2024-03-25T14:33:14.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b0500 con 0x7f5b241430c0 2024-03-25T14:33:14.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240aea50 con 0x7f5b240a98f0 2024-03-25T14:33:14.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:14.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.634+0000 7f5b2bc63640 10 monclient(hunting): authenticate will time out at 1879.736835s 2024-03-25T14:33:14.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.644+0000 7f5b291fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5b2413f710 0x7f5b240aef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:40136/0 (socket says 172.21.15.26:40136) 2024-03-25T14:33:14.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.644+0000 7f5b291fe640 1 -- 172.21.15.26:0/2215731563 learned_addr learned my addr 172.21.15.26:0/2215731563 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:14.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.645+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b2413f710 2024-03-25T14:33:14.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.645+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:14.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.649+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b241430c0 2024-03-25T14:33:14.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.649+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:14.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.652+0000 7f5b227fc640 10 client.?.objecter ms_handle_connect 0x7f5b240a98f0 2024-03-25T14:33:14.657 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.652+0000 7f5b227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:14.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.657+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247433899 0 0) 0x7f5b240b02b0 con 0x7f5b2413f710 2024-03-25T14:33:14.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.657+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:14.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.657+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:14.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.657+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:14.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.657+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4506 2024-03-25T14:33:14.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.657+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10004eb0 con 0x7f5b2413f710 2024-03-25T14:33:14.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.662+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783968986 0 0) 0x7f5b240b0500 con 0x7f5b241430c0 2024-03-25T14:33:14.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.662+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:14.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.662+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:14.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.662+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:14.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.662+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4492 2024-03-25T14:33:14.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.663+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240b02b0 con 0x7f5b241430c0 2024-03-25T14:33:14.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.676+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542802425 0 0) 0x7f5b240aea50 con 0x7f5b240a98f0 2024-03-25T14:33:14.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.676+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:14.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.676+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:14.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.676+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:14.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.676+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4460 2024-03-25T14:33:14.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.676+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240b0500 con 0x7f5b240a98f0 2024-03-25T14:33:14.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4049762930 0 0) 0x7f5b240b02b0 con 0x7f5b241430c0 2024-03-25T14:33:14.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b240aea50 con 0x7f5b241430c0 2024-03-25T14:33:14.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5b1c001e60 con 0x7f5b241430c0 2024-03-25T14:33:14.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:14.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:33:14.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 10 monclient(hunting): dump: 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.686 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.680+0000 7f5b227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.681+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3880517557 0 0) 0x7f5b10004eb0 con 0x7f5b2413f710 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.681+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b240b02b0 con 0x7f5b2413f710 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.681+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5b180019d0 con 0x7f5b2413f710 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.681+0000 7f5b227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.681+0000 7f5b227fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.681+0000 7f5b227fc640 10 monclient(hunting): dump: 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:14.687 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:14.688 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1802762023 0 0) 0x7f5b240aea50 con 0x7f5b241430c0 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 >> v1:172.21.15.26:6790/0 conn(0x7f5b240a98f0 legacy=0x7f5b240ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 >> v1:172.21.15.26:6789/0 conn(0x7f5b2413f710 legacy=0x7f5b240aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 10 monclient: _finish_hunting 0 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 1 monclient: found mon.b 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240af680 con 0x7f5b241430c0 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 10 monclient: _finish_auth 0 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 10 monclient: _check_auth_tickets 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:14.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 5 monclient: authenticate success, global_id 4492 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 10 monclient: _renew_subs 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b24133780 con 0x7f5b241430c0 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 10 monclient: _renew_subs 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240aba90 con 0x7f5b241430c0 2024-03-25T14:33:14.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.700+0000 7f5b2bc63640 1 librados: init done 2024-03-25T14:33:14.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.701+0000 7f5b2bc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:14.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.701+0000 7f5b2bc63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:14.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.701+0000 7f5b2bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:14.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.701+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24133b20 con 0x7f5b241430c0 2024-03-25T14:33:14.984 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b1c002020 con 0x7f5b241430c0 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5b1c003c20 con 0x7f5b241430c0 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b227fc640 10 monclient: dump: 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:14.985 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:14.986 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:14.986 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:14.986 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:14.986 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:14.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.979+0000 7f5b23fff640 4 set_mon_vals no callback set 2024-03-25T14:33:14.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.980+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f5b1c019bf0 con 0x7f5b241430c0 2024-03-25T14:33:14.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.981+0000 7f5b299ff640 10 monclient: get_auth_request con 0x7f5b1006df20 auth_method 0 2024-03-25T14:33:14.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.981+0000 7f5b227fc640 10 client.4492.objecter ms_handle_connect 0x7f5b1006df20 2024-03-25T14:33:14.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.984+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f5b1c053730 con 0x7f5b241430c0 2024-03-25T14:33:14.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.984+0000 7f5b227fc640 10 client.4492.objecter ms_dispatch 0x7f5b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:14.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.984+0000 7f5b227fc640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:14.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.984+0000 7f5b227fc640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:14.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.984+0000 7f5b227fc640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T14:33:14.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.985+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b1c04f240 con 0x7f5b241430c0 2024-03-25T14:33:14.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.985+0000 7f5b227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:14.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:14.985+0000 7f5b227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:15.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.202+0000 7f5b2bc63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:15.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.202+0000 7f5b2bc63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:15.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.202+0000 7f5b2bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:15.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.202+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b24074cc0 con 0x7f5b241430c0 2024-03-25T14:33:15.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.244+0000 7f5b227fc640 1 -- 172.21.15.26:0/2215731563 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f5b1c051170 con 0x7f5b241430c0 2024-03-25T14:33:15.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.244+0000 7f5b227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:15.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.244+0000 7f5b227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:15.249 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:15.249 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 10 librados: watch_flush enter 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 10 librados: watch_flush exit 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 10 client.4492.objecter successfully canceled tick 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 >> v1:172.21.15.121:6800/288813066 conn(0x7f5b1006df20 legacy=0x7f5b100703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 10 monclient: shutdown 2024-03-25T14:33:15.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:15.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.246+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 >> v1:172.21.15.121:6789/0 conn(0x7f5b241430c0 legacy=0x7f5b24133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:15.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.247+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 shutdown_connections 2024-03-25T14:33:15.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.247+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 >> 172.21.15.26:0/2215731563 conn(0x7f5b240b7ea0 msgr2=0x7f5b24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:15.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.247+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 shutdown_connections 2024-03-25T14:33:15.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.247+0000 7f5b2bc63640 1 -- 172.21.15.26:0/2215731563 wait complete. 2024-03-25T14:33:15.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:15.247+0000 7f5b2bc63640 1 librados: shutdown 2024-03-25T14:33:15.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:21.889 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:06:40.749 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.894 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:06:40.755 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.918 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:06:40.778 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.948 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:06:40.809 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.955 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:06:40.815 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.955 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:06:40.832 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.962 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:06:40.839 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.992 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:06:40.868 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:21.998 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:06:40.858 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:22.070 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.174+0000 7fbe96688640 10 monclient: get_monmap_and_config 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.174+0000 7fbe96688640 10 monclient: build_initial_monmap 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.174+0000 7fbe96688640 10 monclient: monmap: 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:22.174887+0000 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:22.174887+0000 2024-03-25T14:33:22.179 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:22.180 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:22.180 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:22.180 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:22.180 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:22.180 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.174+0000 7fbe96688640 1 Processor -- start 2024-03-25T14:33:22.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.174+0000 7fbe96688640 1 -- start start 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.174+0000 7fbe96688640 10 monclient: init 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient(hunting): picked mon.noname-c con 0x7fbe900a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient(hunting): picked mon.noname-b con 0x7fbe9013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient(hunting): picked mon.noname-a con 0x7fbe901430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90142520 con 0x7fbe901430c0 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900b28a0 con 0x7fbe900a98f0 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900b2a30 con 0x7fbe9013f710 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:22.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.175+0000 7fbe96688640 10 monclient(hunting): authenticate will time out at 1887.278198s 2024-03-25T14:33:22.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.187+0000 7fbe94c25640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbe901430c0 0x7fbe9014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55244/0 (socket says 172.21.15.26:55244) 2024-03-25T14:33:22.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.187+0000 7fbe94c25640 1 -- 172.21.15.26:0/407767184 learned_addr learned my addr 172.21.15.26:0/407767184 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:22.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.205+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485650258 0 0) 0x7fbe90142520 con 0x7fbe901430c0 2024-03-25T14:33:22.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.205+0000 7fbe8effd640 10 monclient(hunting): _negotiate 2024-03-25T14:33:22.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.205+0000 7fbe8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:22.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.205+0000 7fbe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:22.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.205+0000 7fbe8effd640 10 monclient(hunting): my global_id is 4509 2024-03-25T14:33:22.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.205+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe78003520 con 0x7fbe901430c0 2024-03-25T14:33:22.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.207+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971160822 0 0) 0x7fbe900b28a0 con 0x7fbe900a98f0 2024-03-25T14:33:22.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.207+0000 7fbe8effd640 10 monclient(hunting): _negotiate 2024-03-25T14:33:22.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.207+0000 7fbe8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:22.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.207+0000 7fbe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:22.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.207+0000 7fbe8effd640 10 monclient(hunting): my global_id is 4495 2024-03-25T14:33:22.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.207+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe90142520 con 0x7fbe900a98f0 2024-03-25T14:33:22.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1482586738 0 0) 0x7fbe78003520 con 0x7fbe901430c0 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 >> v1:172.21.15.26:6790/0 conn(0x7fbe9013f710 legacy=0x7fbe901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 >> v1:172.21.15.121:6789/0 conn(0x7fbe900a98f0 legacy=0x7fbe900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: _finish_hunting 0 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 1 monclient: found mon.noname-a 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900b44c0 con 0x7fbe901430c0 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: _finish_auth 0 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: _check_auth_tickets 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbe88001a10 con 0x7fbe901430c0 2024-03-25T14:33:22.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe96688640 5 monclient: authenticate success, global_id 4509 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe96688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe8effd640 10 monclient: dump: 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:22.228 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:22.229 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.221+0000 7fbe96688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:22.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe88001b70 con 0x7fbe901430c0 2024-03-25T14:33:22.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:22.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe8effd640 1 -- 172.21.15.26:0/407767184 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbe88001f80 con 0x7fbe901430c0 2024-03-25T14:33:22.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:22.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:22.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe8effd640 10 monclient: dump: 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe96688640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:22.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.251+0000 7fbe96688640 4 set_mon_vals no callback set 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 10 monclient: shutdown 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 1 -- 172.21.15.26:0/407767184 >> v1:172.21.15.26:6789/0 conn(0x7fbe901430c0 legacy=0x7fbe9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 1 -- 172.21.15.26:0/407767184 shutdown_connections 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 1 -- 172.21.15.26:0/407767184 >> 172.21.15.26:0/407767184 conn(0x7fbe900b7ea0 msgr2=0x7fbe900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 1 -- 172.21.15.26:0/407767184 shutdown_connections 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.252+0000 7fbe96688640 1 -- 172.21.15.26:0/407767184 wait complete. 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.253+0000 7fbe96688640 10 monclient: build_initial_monmap 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.253+0000 7fbe96688640 10 monclient: monmap: 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:22.259 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.253+0000 7fbe96688640 1 librados: starting msgr at 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.253+0000 7fbe96688640 1 librados: starting objecter 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.253+0000 7fbe96688640 1 Processor -- start 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 1 -- start start 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 1 librados: setting wanted keys 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 1 librados: calling monclient init 2024-03-25T14:33:22.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 10 monclient: init 2024-03-25T14:33:22.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 10 monclient(hunting): picked mon.noname-a con 0x7fbe900a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 10 monclient(hunting): picked mon.noname-b con 0x7fbe9013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 10 monclient(hunting): picked mon.noname-c con 0x7fbe901430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90139830 con 0x7fbe900a98f0 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.254+0000 7fbe96688640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe901db7d0 con 0x7fbe9013f710 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.255+0000 7fbe96688640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe901db410 con 0x7fbe901430c0 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.255+0000 7fbe96688640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:22.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.255+0000 7fbe96688640 10 monclient(hunting): authenticate will time out at 1887.357663s 2024-03-25T14:33:22.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.263+0000 7fbe8ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbe900a98f0 0x7fbe901dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:55294/0 (socket says 172.21.15.26:55294) 2024-03-25T14:33:22.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.263+0000 7fbe8ffff640 1 -- 172.21.15.26:0/3142162650 learned_addr learned my addr 172.21.15.26:0/3142162650 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:22.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.264+0000 7fbe8cff9640 10 client.?.objecter ms_handle_connect 0x7fbe900a98f0 2024-03-25T14:33:22.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.264+0000 7fbe8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:22.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.273+0000 7fbe8cff9640 10 client.?.objecter ms_handle_connect 0x7fbe901430c0 2024-03-25T14:33:22.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.273+0000 7fbe8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:22.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.281+0000 7fbe8cff9640 10 client.?.objecter ms_handle_connect 0x7fbe9013f710 2024-03-25T14:33:22.286 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.281+0000 7fbe8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:22.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.282+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81549924 0 0) 0x7fbe90139830 con 0x7fbe900a98f0 2024-03-25T14:33:22.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.282+0000 7fbe8cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:33:22.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.282+0000 7fbe8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:22.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.282+0000 7fbe8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:22.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.282+0000 7fbe8cff9640 10 monclient(hunting): my global_id is 4512 2024-03-25T14:33:22.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.282+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c004eb0 con 0x7fbe900a98f0 2024-03-25T14:33:22.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.284+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41369471 0 0) 0x7fbe901db410 con 0x7fbe901430c0 2024-03-25T14:33:22.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.284+0000 7fbe8cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:33:22.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.284+0000 7fbe8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:22.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.284+0000 7fbe8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:22.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.284+0000 7fbe8cff9640 10 monclient(hunting): my global_id is 4466 2024-03-25T14:33:22.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.284+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe90139830 con 0x7fbe901430c0 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2432129252 0 0) 0x7fbe90139830 con 0x7fbe901430c0 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe901db410 con 0x7fbe901430c0 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbe88001e80 con 0x7fbe901430c0 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 10 monclient(hunting): dump: 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.298 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.292+0000 7fbe8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389733638 0 0) 0x7fbe7c004eb0 con 0x7fbe900a98f0 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe90139830 con 0x7fbe900a98f0 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbe80001a10 con 0x7fbe900a98f0 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): dump: 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.302 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69138158 0 0) 0x7fbe901db7d0 con 0x7fbe9013f710 2024-03-25T14:33:22.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:33:22.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:22.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:22.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T14:33:22.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.297+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c004eb0 con 0x7fbe9013f710 2024-03-25T14:33:22.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2725907278 0 0) 0x7fbe901db410 con 0x7fbe901430c0 2024-03-25T14:33:22.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 >> v1:172.21.15.121:6789/0 conn(0x7fbe9013f710 legacy=0x7fbe900b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 >> v1:172.21.15.26:6789/0 conn(0x7fbe900a98f0 legacy=0x7fbe901dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 10 monclient: _finish_hunting 0 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 1 monclient: found mon.c 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe901db5a0 con 0x7fbe901430c0 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 10 monclient: _finish_auth 0 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 10 monclient: _check_auth_tickets 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 5 monclient: authenticate success, global_id 4466 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 10 monclient: _renew_subs 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe900b1480 con 0x7fbe901430c0 2024-03-25T14:33:22.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 10 monclient: _renew_subs 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe900b0b10 con 0x7fbe901430c0 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.305+0000 7fbe96688640 1 librados: init done 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.306+0000 7fbe96688640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.306+0000 7fbe96688640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.306+0000 7fbe96688640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:22.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.306+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe900627b0 con 0x7fbe901430c0 2024-03-25T14:33:22.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe880015c0 con 0x7fbe901430c0 2024-03-25T14:33:22.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:22.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbe88004e10 con 0x7fbe901430c0 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 10 monclient: dump: 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:22.629 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:22.630 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:22.630 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:22.630 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:22.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8e7fc640 4 set_mon_vals no callback set 2024-03-25T14:33:22.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.623+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fbe8801ade0 con 0x7fbe901430c0 2024-03-25T14:33:22.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.624+0000 7fbe8ffff640 10 monclient: get_auth_request con 0x7fbe7c06deb0 auth_method 0 2024-03-25T14:33:22.631 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.624+0000 7fbe8cff9640 10 client.4466.objecter ms_handle_connect 0x7fbe7c06deb0 2024-03-25T14:33:22.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fbe88054ab0 con 0x7fbe901430c0 2024-03-25T14:33:22.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 10 client.4466.objecter ms_dispatch 0x7fbe90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:22.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:22.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:22.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T14:33:22.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe880503f0 con 0x7fbe901430c0 2024-03-25T14:33:22.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:22.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.630+0000 7fbe8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:22.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.882+0000 7fbe96688640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:22.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.882+0000 7fbe96688640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:22.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.882+0000 7fbe96688640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:22.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:22.882+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe90069f40 con 0x7fbe901430c0 2024-03-25T14:33:23.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.002+0000 7fbe8cff9640 1 -- 172.21.15.26:0/3142162650 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fbe880524f0 con 0x7fbe901430c0 2024-03-25T14:33:23.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.002+0000 7fbe8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:23.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.002+0000 7fbe8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:23.007 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:23.007 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:23.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 10 librados: watch_flush enter 2024-03-25T14:33:23.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 10 librados: watch_flush exit 2024-03-25T14:33:23.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 10 client.4466.objecter successfully canceled tick 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 >> v1:172.21.15.121:6800/288813066 conn(0x7fbe7c06deb0 legacy=0x7fbe7c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 10 monclient: shutdown 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 >> v1:172.21.15.26:6790/0 conn(0x7fbe901430c0 legacy=0x7fbe901d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 shutdown_connections 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 >> 172.21.15.26:0/3142162650 conn(0x7fbe900b7ea0 msgr2=0x7fbe90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 shutdown_connections 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 -- 172.21.15.26:0/3142162650 wait complete. 2024-03-25T14:33:23.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:23.004+0000 7fbe96688640 1 librados: shutdown 2024-03-25T14:33:23.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:29.828 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:29.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.940+0000 7fc462d90640 10 monclient: get_monmap_and_config 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.940+0000 7fc462d90640 10 monclient: build_initial_monmap 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.940+0000 7fc462d90640 10 monclient: monmap: 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:29.941091+0000 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:29.941091+0000 2024-03-25T14:33:29.946 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:29.947 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:29.947 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:29.947 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:29.947 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:29.947 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:29.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.941+0000 7fc462d90640 1 Processor -- start 2024-03-25T14:33:29.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.941+0000 7fc462d90640 1 -- start start 2024-03-25T14:33:29.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.941+0000 7fc462d90640 10 monclient: init 2024-03-25T14:33:29.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:29.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:33:29.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient(hunting): picked mon.noname-c con 0x7fc45c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:29.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient(hunting): picked mon.noname-a con 0x7fc45c13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:29.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient(hunting): picked mon.noname-b con 0x7fc45c1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:29.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c142520 con 0x7fc45c13f710 2024-03-25T14:33:29.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b28a0 con 0x7fc45c0a98f0 2024-03-25T14:33:29.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b2a30 con 0x7fc45c1430c0 2024-03-25T14:33:29.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:29.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.942+0000 7fc462d90640 10 monclient(hunting): authenticate will time out at 1895.044856s 2024-03-25T14:33:29.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.945+0000 7fc45bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc45c13f710 0x7fc45c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47396/0 (socket says 172.21.15.26:47396) 2024-03-25T14:33:29.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.945+0000 7fc45bfff640 1 -- 172.21.15.26:0/3695564556 learned_addr learned my addr 172.21.15.26:0/3695564556 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:29.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.954+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73000227 0 0) 0x7fc45c0b28a0 con 0x7fc45c0a98f0 2024-03-25T14:33:29.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.954+0000 7fc45b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:33:29.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.954+0000 7fc45b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:29.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.954+0000 7fc45b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:29.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.954+0000 7fc45b7fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T14:33:29.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.954+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c003520 con 0x7fc45c0a98f0 2024-03-25T14:33:29.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37785607 0 0) 0x7fc45c142520 con 0x7fc45c13f710 2024-03-25T14:33:29.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c0b28a0 con 0x7fc45c13f710 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.961+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469297149 0 0) 0x7fc44c003520 con 0x7fc45c0a98f0 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 >> v1:172.21.15.26:6790/0 conn(0x7fc45c1430c0 legacy=0x7fc45c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 >> v1:172.21.15.26:6789/0 conn(0x7fc45c13f710 legacy=0x7fc45c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 1 monclient: found mon.noname-c 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0b44c0 con 0x7fc45c0a98f0 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:33:29.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc450001690 con 0x7fc45c0a98f0 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc45b7fe640 10 monclient: dump: 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:29.969 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc462d90640 5 monclient: authenticate success, global_id 4501 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.962+0000 7fc462d90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4500017f0 con 0x7fc45c0a98f0 2024-03-25T14:33:29.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc45b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:29.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc462d90640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:29.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc462d90640 4 set_mon_vals no callback set 2024-03-25T14:33:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc45b7fe640 1 -- 172.21.15.26:0/3695564556 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc450001c00 con 0x7fc45c0a98f0 2024-03-25T14:33:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc45b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:29.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc45b7fe640 10 monclient: dump: 2024-03-25T14:33:29.973 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:29.973 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc462d90640 10 monclient: shutdown 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc462d90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.966+0000 7fc462d90640 1 -- 172.21.15.26:0/3695564556 >> v1:172.21.15.121:6789/0 conn(0x7fc45c0a98f0 legacy=0x7fc45c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.967+0000 7fc462d90640 1 -- 172.21.15.26:0/3695564556 shutdown_connections 2024-03-25T14:33:29.974 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.967+0000 7fc462d90640 1 -- 172.21.15.26:0/3695564556 >> 172.21.15.26:0/3695564556 conn(0x7fc45c0b7ea0 msgr2=0x7fc45c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.967+0000 7fc462d90640 1 -- 172.21.15.26:0/3695564556 shutdown_connections 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.967+0000 7fc462d90640 1 -- 172.21.15.26:0/3695564556 wait complete. 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 10 monclient: build_initial_monmap 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 10 monclient: monmap: 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:29.975 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 1 librados: starting msgr at 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 1 librados: starting objecter 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 1 Processor -- start 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 1 -- start start 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 1 librados: setting wanted keys 2024-03-25T14:33:29.976 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 1 librados: calling monclient init 2024-03-25T14:33:29.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.968+0000 7fc462d90640 10 monclient: init 2024-03-25T14:33:29.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:29.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:33:29.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient(hunting): picked mon.noname-a con 0x7fc45c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient(hunting): picked mon.noname-c con 0x7fc45c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient(hunting): picked mon.noname-b con 0x7fc45c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c1d78d0 con 0x7fc45c0a98f0 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c137f30 con 0x7fc45c1430c0 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c139090 con 0x7fc45c13f710 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:29.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.969+0000 7fc462d90640 10 monclient(hunting): authenticate will time out at 1895.072000s 2024-03-25T14:33:29.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.976+0000 7fc46132d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc45c1430c0 0x7fc45c1d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:40004/0 (socket says 172.21.15.26:40004) 2024-03-25T14:33:29.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.976+0000 7fc46132d640 1 -- 172.21.15.26:0/1588467156 learned_addr learned my addr 172.21.15.26:0/1588467156 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:29.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.980+0000 7fc4597fa640 10 client.?.objecter ms_handle_connect 0x7fc45c1430c0 2024-03-25T14:33:29.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.980+0000 7fc4597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:29.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.984+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420357505 0 0) 0x7fc45c137f30 con 0x7fc45c1430c0 2024-03-25T14:33:29.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.984+0000 7fc4597fa640 10 monclient(hunting): _negotiate 2024-03-25T14:33:29.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.984+0000 7fc4597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:29.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.984+0000 7fc4597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:29.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.984+0000 7fc4597fa640 10 monclient(hunting): my global_id is 4504 2024-03-25T14:33:29.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.984+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc440004eb0 con 0x7fc45c1430c0 2024-03-25T14:33:30.004 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.999+0000 7fc4597fa640 10 client.?.objecter ms_handle_connect 0x7fc45c13f710 2024-03-25T14:33:30.004 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:29.999+0000 7fc4597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:30.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.000+0000 7fc4597fa640 10 client.?.objecter ms_handle_connect 0x7fc45c0a98f0 2024-03-25T14:33:30.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.000+0000 7fc4597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:30.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.008+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772863164 0 0) 0x7fc45c139090 con 0x7fc45c13f710 2024-03-25T14:33:30.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.008+0000 7fc4597fa640 10 monclient(hunting): _negotiate 2024-03-25T14:33:30.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.008+0000 7fc4597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:30.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.008+0000 7fc4597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:30.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.008+0000 7fc4597fa640 10 monclient(hunting): my global_id is 4469 2024-03-25T14:33:30.013 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.008+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c137f30 con 0x7fc45c13f710 2024-03-25T14:33:30.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1644139133 0 0) 0x7fc440004eb0 con 0x7fc45c1430c0 2024-03-25T14:33:30.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc45c139090 con 0x7fc45c1430c0 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc4540042e0 con 0x7fc45c1430c0 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 10 monclient(hunting): dump: 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:30.015 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:30.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.009+0000 7fc4597fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:30.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.014+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397780381 0 0) 0x7fc45c1d78d0 con 0x7fc45c0a98f0 2024-03-25T14:33:30.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.014+0000 7fc4597fa640 10 monclient(hunting): _negotiate 2024-03-25T14:33:30.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.014+0000 7fc4597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:30.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.014+0000 7fc4597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:30.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.014+0000 7fc4597fa640 10 monclient(hunting): my global_id is 4518 2024-03-25T14:33:30.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.014+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc440004eb0 con 0x7fc45c0a98f0 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.022+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3079203652 0 0) 0x7fc45c137f30 con 0x7fc45c13f710 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.022+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc45c1d78d0 con 0x7fc45c13f710 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.022+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc4480018b0 con 0x7fc45c13f710 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.022+0000 7fc4597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.022+0000 7fc4597fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.022+0000 7fc4597fa640 10 monclient(hunting): dump: 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:30.027 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:30.028 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:30.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2713097342 0 0) 0x7fc45c139090 con 0x7fc45c1430c0 2024-03-25T14:33:30.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 >> v1:172.21.15.26:6790/0 conn(0x7fc45c13f710 legacy=0x7fc45c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:30.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 >> v1:172.21.15.26:6789/0 conn(0x7fc45c0a98f0 legacy=0x7fc45c138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:30.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 10 monclient: _finish_hunting 0 2024-03-25T14:33:30.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 1 monclient: found mon.b 2024-03-25T14:33:30.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c1d8500 con 0x7fc45c1430c0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 10 monclient: _finish_auth 0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 10 monclient: _check_auth_tickets 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc4597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 5 monclient: authenticate success, global_id 4504 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 10 monclient: _renew_subs 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c139370 con 0x7fc45c1430c0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 10 monclient: _renew_subs 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c1d43c0 con 0x7fc45c1430c0 2024-03-25T14:33:30.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.025+0000 7fc462d90640 1 librados: init done 2024-03-25T14:33:30.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.026+0000 7fc462d90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:30.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.026+0000 7fc462d90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:30.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.026+0000 7fc462d90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:30.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.026+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c0627b0 con 0x7fc45c1430c0 2024-03-25T14:33:30.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.244+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc454001250 con 0x7fc45c1430c0 2024-03-25T14:33:30.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.244+0000 7fc4597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:30.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.244+0000 7fc45affd640 4 set_mon_vals no callback set 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.294+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc454001660 con 0x7fc45c1430c0 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.294+0000 7fc4597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.294+0000 7fc4597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.294+0000 7fc4597fa640 10 monclient: dump: 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:30.299 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:30.300 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:30.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.295+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fc454001840 con 0x7fc45c1430c0 2024-03-25T14:33:30.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.295+0000 7fc460b2c640 10 monclient: get_auth_request con 0x7fc44006df60 auth_method 0 2024-03-25T14:33:30.301 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.295+0000 7fc4597fa640 10 client.4504.objecter ms_handle_connect 0x7fc44006df60 2024-03-25T14:33:30.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.298+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fc454054a40 con 0x7fc45c1430c0 2024-03-25T14:33:30.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.298+0000 7fc4597fa640 10 client.4504.objecter ms_dispatch 0x7fc45c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:30.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.298+0000 7fc4597fa640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:30.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.298+0000 7fc4597fa640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:30.303 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.298+0000 7fc4597fa640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T14:33:30.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.302+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc454050550 con 0x7fc45c1430c0 2024-03-25T14:33:30.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.302+0000 7fc4597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:30.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.302+0000 7fc4597fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:30.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.543+0000 7fc462d90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:30.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.543+0000 7fc462d90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:30.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.543+0000 7fc462d90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:30.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.543+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc45c1d8690 con 0x7fc45c1430c0 2024-03-25T14:33:30.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.565+0000 7fc4597fa640 1 -- 172.21.15.26:0/1588467156 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fc454052480 con 0x7fc45c1430c0 2024-03-25T14:33:30.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.565+0000 7fc4597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:30.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.565+0000 7fc4597fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:30.571 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:30.571 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:30.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 10 librados: watch_flush enter 2024-03-25T14:33:30.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 10 librados: watch_flush exit 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 10 client.4504.objecter successfully canceled tick 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 >> v1:172.21.15.121:6800/288813066 conn(0x7fc44006df60 legacy=0x7fc440070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 10 monclient: shutdown 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 >> v1:172.21.15.121:6789/0 conn(0x7fc45c1430c0 legacy=0x7fc45c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 shutdown_connections 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 >> 172.21.15.26:0/1588467156 conn(0x7fc45c0b7ea0 msgr2=0x7fc45c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 shutdown_connections 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 -- 172.21.15.26:0/1588467156 wait complete. 2024-03-25T14:33:30.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:30.568+0000 7fc462d90640 1 librados: shutdown 2024-03-25T14:33:30.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:37.393 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:37.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.504+0000 7fc002b9d640 10 monclient: get_monmap_and_config 2024-03-25T14:33:37.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.504+0000 7fc002b9d640 10 monclient: build_initial_monmap 2024-03-25T14:33:37.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.504+0000 7fc002b9d640 10 monclient: monmap: 2024-03-25T14:33:37.509 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:37.509 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:37.509 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:37.505133+0000 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:37.505133+0000 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:37.510 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.505+0000 7fc002b9d640 1 Processor -- start 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.505+0000 7fc002b9d640 1 -- start start 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.505+0000 7fc002b9d640 10 monclient: init 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fbffc0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fbffc13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fbffc1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc142520 con 0x7fbffc13f710 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc0b28a0 con 0x7fbffc0a98f0 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc0b2a30 con 0x7fbffc1430c0 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:37.512 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.506+0000 7fc002b9d640 10 monclient(hunting): authenticate will time out at 1902.609076s 2024-03-25T14:33:37.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.517+0000 7fc00113a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbffc1430c0 0x7fbffc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54968/0 (socket says 172.21.15.26:54968) 2024-03-25T14:33:37.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.517+0000 7fc00113a640 1 -- 172.21.15.26:0/888713598 learned_addr learned my addr 172.21.15.26:0/888713598 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:37.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.529+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811744560 0 0) 0x7fbffc0b2a30 con 0x7fbffc1430c0 2024-03-25T14:33:37.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.529+0000 7fbffb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:33:37.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.529+0000 7fbffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:37.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.529+0000 7fbffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:37.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.529+0000 7fbffb7fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T14:33:37.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.529+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfec003520 con 0x7fbffc1430c0 2024-03-25T14:33:37.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2569382018 0 0) 0x7fbfec003520 con 0x7fbffc1430c0 2024-03-25T14:33:37.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 >> v1:172.21.15.121:6789/0 conn(0x7fbffc0a98f0 legacy=0x7fbffc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:37.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 >> v1:172.21.15.26:6789/0 conn(0x7fbffc13f710 legacy=0x7fbffc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 1 monclient: found mon.noname-b 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc0b44c0 con 0x7fbffc1430c0 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: _finish_auth 0 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbff4001a10 con 0x7fbffc1430c0 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fbffb7fe640 10 monclient: dump: 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:37.548 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fc002b9d640 5 monclient: authenticate success, global_id 4472 2024-03-25T14:33:37.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.542+0000 7fc002b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbff4001b70 con 0x7fbffc1430c0 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fbffb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fbffb7fe640 1 -- 172.21.15.26:0/888713598 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbff4001fc0 con 0x7fbffc1430c0 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fbffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fbffb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fbffb7fe640 10 monclient: dump: 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:37.559 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:37.560 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:37.560 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:37.560 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fc002b9d640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:37.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fc002b9d640 4 set_mon_vals no callback set 2024-03-25T14:33:37.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fc002b9d640 10 monclient: shutdown 2024-03-25T14:33:37.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fc002b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:37.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.554+0000 7fc002b9d640 1 -- 172.21.15.26:0/888713598 >> v1:172.21.15.26:6790/0 conn(0x7fbffc1430c0 legacy=0x7fbffc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.555+0000 7fc002b9d640 1 -- 172.21.15.26:0/888713598 shutdown_connections 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.555+0000 7fc002b9d640 1 -- 172.21.15.26:0/888713598 >> 172.21.15.26:0/888713598 conn(0x7fbffc0b7ea0 msgr2=0x7fbffc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.555+0000 7fc002b9d640 1 -- 172.21.15.26:0/888713598 shutdown_connections 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.555+0000 7fc002b9d640 1 -- 172.21.15.26:0/888713598 wait complete. 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.555+0000 7fc002b9d640 10 monclient: build_initial_monmap 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.555+0000 7fc002b9d640 10 monclient: monmap: 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:37.562 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 1 librados: starting msgr at 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 1 librados: starting objecter 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 1 Processor -- start 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 1 -- start start 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 1 librados: setting wanted keys 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 1 librados: calling monclient init 2024-03-25T14:33:37.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.556+0000 7fc002b9d640 10 monclient: init 2024-03-25T14:33:37.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:37.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:33:37.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fbffc0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fbffc13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fbffc1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc0abd30 con 0x7fbffc0a98f0 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc1d7810 con 0x7fbffc13f710 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc1d8440 con 0x7fbffc1430c0 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:37.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.557+0000 7fc002b9d640 10 monclient(hunting): authenticate will time out at 1902.660097s 2024-03-25T14:33:37.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.567+0000 7fc000939640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbffc0a98f0 0x7fbffc1d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52044/0 (socket says 172.21.15.26:52044) 2024-03-25T14:33:37.572 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.567+0000 7fc000939640 1 -- 172.21.15.26:0/2195298038 learned_addr learned my addr 172.21.15.26:0/2195298038 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:37.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.573+0000 7fbff97fa640 10 client.?.objecter ms_handle_connect 0x7fbffc1430c0 2024-03-25T14:33:37.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.573+0000 7fbff97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:37.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.574+0000 7fbff97fa640 10 client.?.objecter ms_handle_connect 0x7fbffc0a98f0 2024-03-25T14:33:37.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.574+0000 7fbff97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:37.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.585+0000 7fbff97fa640 10 client.?.objecter ms_handle_connect 0x7fbffc13f710 2024-03-25T14:33:37.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.585+0000 7fbff97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:37.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.590+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312469217 0 0) 0x7fbffc0abd30 con 0x7fbffc0a98f0 2024-03-25T14:33:37.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.590+0000 7fbff97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:33:37.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.590+0000 7fbff97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:37.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.590+0000 7fbff97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:37.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.590+0000 7fbff97fa640 10 monclient(hunting): my global_id is 4524 2024-03-25T14:33:37.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.590+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe0004eb0 con 0x7fbffc0a98f0 2024-03-25T14:33:37.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.592+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540793185 0 0) 0x7fbffc1d8440 con 0x7fbffc1430c0 2024-03-25T14:33:37.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.592+0000 7fbff97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:33:37.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.592+0000 7fbff97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:37.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.592+0000 7fbff97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:37.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.592+0000 7fbff97fa640 10 monclient(hunting): my global_id is 4475 2024-03-25T14:33:37.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.592+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbffc0abd30 con 0x7fbffc1430c0 2024-03-25T14:33:37.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.603+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141098031 0 0) 0x7fbffc1d7810 con 0x7fbffc13f710 2024-03-25T14:33:37.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.603+0000 7fbff97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:33:37.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.603+0000 7fbff97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:37.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.603+0000 7fbff97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:37.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.603+0000 7fbff97fa640 10 monclient(hunting): my global_id is 4510 2024-03-25T14:33:37.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.603+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbffc1d8440 con 0x7fbffc13f710 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.608+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2588607630 0 0) 0x7fbfe0004eb0 con 0x7fbffc0a98f0 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.609+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbffc1d7810 con 0x7fbffc0a98f0 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.609+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbff00019d0 con 0x7fbffc0a98f0 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.609+0000 7fbff97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.609+0000 7fbff97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.609+0000 7fbff97fa640 10 monclient(hunting): dump: 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.614 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:37.615 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:37.615 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:37.615 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:37.615 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:37.615 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.615 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.609+0000 7fbff97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.612+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1828111170 0 0) 0x7fbffc0abd30 con 0x7fbffc1430c0 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.612+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe0004eb0 con 0x7fbffc1430c0 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.612+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbff4001e80 con 0x7fbffc1430c0 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.612+0000 7fbff97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.612+0000 7fbff97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.612+0000 7fbff97fa640 10 monclient(hunting): dump: 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:37.617 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:37.618 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3548359816 0 0) 0x7fbfe0004eb0 con 0x7fbffc1430c0 2024-03-25T14:33:37.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 >> v1:172.21.15.121:6789/0 conn(0x7fbffc13f710 legacy=0x7fbffc1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:37.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 >> v1:172.21.15.26:6789/0 conn(0x7fbffc0a98f0 legacy=0x7fbffc1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:37.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 10 monclient: _finish_hunting 0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 1 monclient: found mon.c 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc1d5000 con 0x7fbffc1430c0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 10 monclient: _finish_auth 0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 10 monclient: _check_auth_tickets 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fbff97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 5 monclient: authenticate success, global_id 4475 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 10 monclient: _renew_subs 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc1d5250 con 0x7fbffc1430c0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 10 monclient: _renew_subs 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:37.629 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc1d43c0 con 0x7fbffc1430c0 2024-03-25T14:33:37.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.623+0000 7fc002b9d640 1 librados: init done 2024-03-25T14:33:37.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.624+0000 7fc002b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:37.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.624+0000 7fc002b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:37.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.624+0000 7fc002b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:37.630 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.624+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc1d7670 con 0x7fbffc1430c0 2024-03-25T14:33:37.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.627+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbff4002040 con 0x7fbffc1430c0 2024-03-25T14:33:37.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.627+0000 7fbff97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:37.632 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.627+0000 7fbffaffd640 4 set_mon_vals no callback set 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.951+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fbff4004e10 con 0x7fbffc1430c0 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.951+0000 7fbff97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.951+0000 7fbff97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.951+0000 7fbff97fa640 10 monclient: dump: 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:37.957 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:37.958 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:37.958 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:37.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.952+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fbff401ade0 con 0x7fbffc1430c0 2024-03-25T14:33:37.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.953+0000 7fc000939640 10 monclient: get_auth_request con 0x7fbfe006dfb0 auth_method 0 2024-03-25T14:33:37.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.953+0000 7fbff97fa640 10 client.4475.objecter ms_handle_connect 0x7fbfe006dfb0 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fbff4054960 con 0x7fbffc1430c0 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 10 client.4475.objecter ms_dispatch 0x7fbffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbff4050470 con 0x7fbffc1430c0 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:37.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:37.960+0000 7fbff97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:38.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.189+0000 7fc002b9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:38.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.189+0000 7fc002b9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:38.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.190+0000 7fc002b9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:33:38.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.190+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbffc0abb70 con 0x7fbffc1430c0 2024-03-25T14:33:38.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.215+0000 7fbff97fa640 1 -- 172.21.15.26:0/2195298038 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fbff40523a0 con 0x7fbffc1430c0 2024-03-25T14:33:38.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.215+0000 7fbff97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:38.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.215+0000 7fbff97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:38.221 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:38.221 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 10 librados: watch_flush enter 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 10 librados: watch_flush exit 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 10 client.4475.objecter successfully canceled tick 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 >> v1:172.21.15.121:6800/288813066 conn(0x7fbfe006dfb0 legacy=0x7fbfe0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 10 monclient: shutdown 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:38.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 >> v1:172.21.15.26:6790/0 conn(0x7fbffc1430c0 legacy=0x7fbffc1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:38.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 shutdown_connections 2024-03-25T14:33:38.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.217+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 >> 172.21.15.26:0/2195298038 conn(0x7fbffc0b7ea0 msgr2=0x7fbffc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:38.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.218+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 shutdown_connections 2024-03-25T14:33:38.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.218+0000 7fc002b9d640 1 -- 172.21.15.26:0/2195298038 wait complete. 2024-03-25T14:33:38.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:38.218+0000 7fc002b9d640 1 librados: shutdown 2024-03-25T14:33:38.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:41.971 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:00.831 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:41.981 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:00.841 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.006 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:00.867 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.012 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:00.888 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.016 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:00.892 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.034 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:00.910 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.037 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:00.897 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.041 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:00.902 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:42.079 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:00.939 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:33:45.043 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.152+0000 7fe9c398f640 10 monclient: get_monmap_and_config 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.152+0000 7fe9c398f640 10 monclient: build_initial_monmap 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.152+0000 7fe9c398f640 10 monclient: monmap: 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:45.152885+0000 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:45.152885+0000 2024-03-25T14:33:45.157 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:45.158 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:45.158 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:45.158 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:45.158 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:45.158 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.159 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.152+0000 7fe9c398f640 1 Processor -- start 2024-03-25T14:33:45.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.152+0000 7fe9c398f640 1 -- start start 2024-03-25T14:33:45.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.152+0000 7fe9c398f640 10 monclient: init 2024-03-25T14:33:45.160 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient(hunting): picked mon.noname-b con 0x7fe9bc0a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient(hunting): picked mon.noname-c con 0x7fe9bc13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient(hunting): picked mon.noname-a con 0x7fe9bc1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc142520 con 0x7fe9bc1430c0 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc0b28a0 con 0x7fe9bc13f710 2024-03-25T14:33:45.161 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc0b2a30 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:45.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.153+0000 7fe9c398f640 10 monclient(hunting): authenticate will time out at 1910.256365s 2024-03-25T14:33:45.168 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.163+0000 7fe9c172b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe9bc0a98f0 0x7fe9bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52210/0 (socket says 172.21.15.26:52210) 2024-03-25T14:33:45.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.163+0000 7fe9c172b640 1 -- 172.21.15.26:0/1175311102 learned_addr learned my addr 172.21.15.26:0/1175311102 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:45.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.174+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907300363 0 0) 0x7fe9bc0b2a30 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.174+0000 7fe9b3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:45.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.174+0000 7fe9b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:45.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.174+0000 7fe9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:45.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.174+0000 7fe9b3fff640 10 monclient(hunting): my global_id is 4478 2024-03-25T14:33:45.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.174+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a8003520 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.179+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452833003 0 0) 0x7fe9bc142520 con 0x7fe9bc1430c0 2024-03-25T14:33:45.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.179+0000 7fe9b3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:45.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.179+0000 7fe9b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:45.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.179+0000 7fe9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:45.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.179+0000 7fe9b3fff640 10 monclient(hunting): my global_id is 4527 2024-03-25T14:33:45.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.179+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc0b2a30 con 0x7fe9bc1430c0 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412555208 0 0) 0x7fe9a8003520 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 >> v1:172.21.15.121:6789/0 conn(0x7fe9bc13f710 legacy=0x7fe9bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 >> v1:172.21.15.26:6789/0 conn(0x7fe9bc1430c0 legacy=0x7fe9bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: _finish_hunting 0 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 1 monclient: found mon.noname-b 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:33:45.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0b44c0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: _finish_auth 0 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: _check_auth_tickets 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9c398f640 5 monclient: authenticate success, global_id 4478 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9c398f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe9b8001690 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9b3fff640 10 monclient: dump: 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.191 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.184+0000 7fe9c398f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b80017f0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9b3fff640 1 -- 172.21.15.26:0/1175311102 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe9b8001c00 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9b3fff640 10 monclient: dump: 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:45.207 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:45.208 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:45.208 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:45.208 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:45.208 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9c398f640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:45.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.201+0000 7fe9c398f640 4 set_mon_vals no callback set 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 10 monclient: shutdown 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 1 -- 172.21.15.26:0/1175311102 >> v1:172.21.15.26:6790/0 conn(0x7fe9bc0a98f0 legacy=0x7fe9bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 1 -- 172.21.15.26:0/1175311102 shutdown_connections 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 1 -- 172.21.15.26:0/1175311102 >> 172.21.15.26:0/1175311102 conn(0x7fe9bc0b7ea0 msgr2=0x7fe9bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 1 -- 172.21.15.26:0/1175311102 shutdown_connections 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.202+0000 7fe9c398f640 1 -- 172.21.15.26:0/1175311102 wait complete. 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.203+0000 7fe9c398f640 10 monclient: build_initial_monmap 2024-03-25T14:33:45.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.203+0000 7fe9c398f640 10 monclient: monmap: 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 librados: starting msgr at 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 librados: starting objecter 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 Processor -- start 2024-03-25T14:33:45.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 -- start start 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 librados: setting wanted keys 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 librados: calling monclient init 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient: init 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient(hunting): picked mon.noname-b con 0x7fe9bc0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient(hunting): picked mon.noname-a con 0x7fe9bc13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient(hunting): picked mon.noname-c con 0x7fe9bc1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc0703b0 con 0x7fe9bc13f710 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc1def30 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc06dfc0 con 0x7fe9bc1430c0 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:45.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.204+0000 7fe9c398f640 10 monclient(hunting): authenticate will time out at 1910.307373s 2024-03-25T14:33:45.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.210+0000 7fe9c172b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe9bc0a98f0 0x7fe9bc1df450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:37104/0 (socket says 172.21.15.26:37104) 2024-03-25T14:33:45.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.210+0000 7fe9c172b640 1 -- 172.21.15.26:0/2995522624 learned_addr learned my addr 172.21.15.26:0/2995522624 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:45.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.210+0000 7fe9b1ffb640 10 client.?.objecter ms_handle_connect 0x7fe9bc0a98f0 2024-03-25T14:33:45.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.210+0000 7fe9b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:45.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.211+0000 7fe9b1ffb640 10 client.?.objecter ms_handle_connect 0x7fe9bc13f710 2024-03-25T14:33:45.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.211+0000 7fe9b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:45.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.217+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058003372 0 0) 0x7fe9bc1def30 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.217+0000 7fe9b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:33:45.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.217+0000 7fe9b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:45.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.217+0000 7fe9b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:45.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.217+0000 7fe9b1ffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T14:33:45.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.217+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a0004eb0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.224+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023022645 0 0) 0x7fe9bc0703b0 con 0x7fe9bc13f710 2024-03-25T14:33:45.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.224+0000 7fe9b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:33:45.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.224+0000 7fe9b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:45.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.224+0000 7fe9b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:45.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.224+0000 7fe9b1ffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T14:33:45.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.224+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc1def30 con 0x7fe9bc13f710 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437474682 0 0) 0x7fe9a0004eb0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9bc0703b0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe9b8001bc0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 10 monclient(hunting): dump: 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:45.236 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:45.237 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:45.237 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:45.237 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:45.237 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.231+0000 7fe9b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:45.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.237+0000 7fe9b1ffb640 10 client.?.objecter ms_handle_connect 0x7fe9bc1430c0 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.237+0000 7fe9b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.239+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3184901752 0 0) 0x7fe9bc1def30 con 0x7fe9bc13f710 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.239+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0004eb0 con 0x7fe9bc13f710 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.239+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe9ac001930 con 0x7fe9bc13f710 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.239+0000 7fe9b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.239+0000 7fe9b1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.239+0000 7fe9b1ffb640 10 monclient(hunting): dump: 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.247 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:45.248 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:45.248 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:45.248 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:45.248 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:45.248 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.250+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236162017 0 0) 0x7fe9bc06dfc0 con 0x7fe9bc1430c0 2024-03-25T14:33:45.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.250+0000 7fe9b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:33:45.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.250+0000 7fe9b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:45.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.250+0000 7fe9b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:45.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.250+0000 7fe9b1ffb640 10 monclient(hunting): my global_id is 4481 2024-03-25T14:33:45.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.250+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc1def30 con 0x7fe9bc1430c0 2024-03-25T14:33:45.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3968772715 0 0) 0x7fe9bc0703b0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 >> v1:172.21.15.26:6790/0 conn(0x7fe9bc1430c0 legacy=0x7fe9bc138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 >> v1:172.21.15.26:6789/0 conn(0x7fe9bc13f710 legacy=0x7fe9bc06eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 1 monclient: found mon.b 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc1dfb60 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 10 monclient: _finish_auth 0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9c398f640 5 monclient: authenticate success, global_id 4516 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9c398f640 10 monclient: _renew_subs 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9c398f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.256+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc06f1b0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.257+0000 7fe9c398f640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.257+0000 7fe9c398f640 10 monclient: _renew_subs 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.257+0000 7fe9c398f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:45.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.257+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc1398e0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.257+0000 7fe9c398f640 1 librados: init done 2024-03-25T14:33:45.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.258+0000 7fe9c398f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:45.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.258+0000 7fe9c398f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:45.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.258+0000 7fe9c398f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:45.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.258+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc1df0c0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b80011f0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:45.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe9b8004cb0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:45.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:45.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b1ffb640 10 monclient: dump: 2024-03-25T14:33:45.542 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.536+0000 7fe9b37fe640 4 set_mon_vals no callback set 2024-03-25T14:33:45.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.537+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fe9b801ad00 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.537+0000 7fe9c0f2a640 10 monclient: get_auth_request con 0x7fe9a006deb0 auth_method 0 2024-03-25T14:33:45.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.537+0000 7fe9b1ffb640 10 client.4516.objecter ms_handle_connect 0x7fe9a006deb0 2024-03-25T14:33:45.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.542+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fe9b8054990 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.542+0000 7fe9b1ffb640 10 client.4516.objecter ms_dispatch 0x7fe9bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:45.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.542+0000 7fe9b1ffb640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:45.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.542+0000 7fe9b1ffb640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:45.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.542+0000 7fe9b1ffb640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T14:33:45.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.543+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9b80502d0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.543+0000 7fe9b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:45.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.543+0000 7fe9b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:45.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.774+0000 7fe9c398f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:45.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.774+0000 7fe9c398f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:45.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.774+0000 7fe9c398f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:45.779 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.774+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9bc1dfcf0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.866 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.861+0000 7fe9b1ffb640 1 -- 172.21.15.26:0/2995522624 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fe9b80523d0 con 0x7fe9bc0a98f0 2024-03-25T14:33:45.867 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.861+0000 7fe9b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:45.867 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.861+0000 7fe9b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:45.867 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:45.867 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 10 librados: watch_flush enter 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 10 librados: watch_flush exit 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 10 client.4516.objecter successfully canceled tick 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 >> v1:172.21.15.121:6800/288813066 conn(0x7fe9a006deb0 legacy=0x7fe9a0070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 10 monclient: shutdown 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 >> v1:172.21.15.121:6789/0 conn(0x7fe9bc0a98f0 legacy=0x7fe9bc1df450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 shutdown_connections 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 >> 172.21.15.26:0/2995522624 conn(0x7fe9bc0b7ea0 msgr2=0x7fe9bc14e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:45.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 shutdown_connections 2024-03-25T14:33:45.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 -- 172.21.15.26:0/2995522624 wait complete. 2024-03-25T14:33:45.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:45.863+0000 7fe9c398f640 1 librados: shutdown 2024-03-25T14:33:45.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:33:52.688 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.907+0000 7fe38e154640 10 monclient: get_monmap_and_config 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.907+0000 7fe38e154640 10 monclient: build_initial_monmap 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.907+0000 7fe38e154640 10 monclient: monmap: 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:33:52.909002+0000 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:33:52.909002+0000 2024-03-25T14:33:52.913 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:52.914 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:52.914 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:52.914 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:33:52.914 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:33:52.914 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.908+0000 7fe38e154640 1 Processor -- start 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.908+0000 7fe38e154640 1 -- start start 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.908+0000 7fe38e154640 10 monclient: init 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.908+0000 7fe38e154640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.908+0000 7fe38e154640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 10 monclient(hunting): picked mon.noname-c con 0x7fe3880b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 10 monclient(hunting): picked mon.noname-b con 0x7fe3880b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 10 monclient(hunting): picked mon.noname-a con 0x7fe38814be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880af940 con 0x7fe38814be90 2024-03-25T14:33:52.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880b0a70 con 0x7fe3880b4d60 2024-03-25T14:33:52.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3880b0d50 con 0x7fe3880b1730 2024-03-25T14:33:52.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:52.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.909+0000 7fe38e154640 10 monclient(hunting): authenticate will time out at 1918.012701s 2024-03-25T14:33:52.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.915+0000 7fe386ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe3880b1730 0x7fe3880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57438/0 (socket says 172.21.15.26:57438) 2024-03-25T14:33:52.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.915+0000 7fe386ffd640 1 -- 172.21.15.26:0/2438120597 learned_addr learned my addr 172.21.15.26:0/2438120597 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:52.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.928+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527746492 0 0) 0x7fe3880b0d50 con 0x7fe3880b1730 2024-03-25T14:33:52.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.928+0000 7fe3867fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:52.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.928+0000 7fe3867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:52.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.928+0000 7fe3867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:52.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.928+0000 7fe3867fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T14:33:52.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.928+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe370003520 con 0x7fe3880b1730 2024-03-25T14:33:52.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.943+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229330433 0 0) 0x7fe3880b0a70 con 0x7fe3880b4d60 2024-03-25T14:33:52.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.943+0000 7fe3867fc640 10 monclient(hunting): _negotiate 2024-03-25T14:33:52.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.943+0000 7fe3867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:52.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.943+0000 7fe3867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:52.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.943+0000 7fe3867fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T14:33:52.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.943+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3880b0d50 con 0x7fe3880b4d60 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.947+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1882541126 0 0) 0x7fe370003520 con 0x7fe3880b1730 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.947+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 >> v1:172.21.15.121:6789/0 conn(0x7fe3880b4d60 legacy=0x7fe38813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.947+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 >> v1:172.21.15.26:6789/0 conn(0x7fe38814be90 legacy=0x7fe38814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.947+0000 7fe3867fc640 10 monclient: _finish_hunting 0 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.947+0000 7fe3867fc640 1 monclient: found mon.noname-b 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880b1030 con 0x7fe3880b1730 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 10 monclient: _finish_auth 0 2024-03-25T14:33:52.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 10 monclient: _check_auth_tickets 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe37c001690 con 0x7fe3880b1730 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe38e154640 5 monclient: authenticate success, global_id 4484 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe38e154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe3867fc640 10 monclient: dump: 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:52.955 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:52.956 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:52.956 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:52.956 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:52.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.948+0000 7fe38e154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:33:52.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe37c0017f0 con 0x7fe3880b1730 2024-03-25T14:33:52.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe3867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:52.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe3867fc640 1 -- 172.21.15.26:0/2438120597 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe37c001c00 con 0x7fe3880b1730 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe38e154640 10 monclient: get_monmap_and_config success 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe38e154640 4 set_mon_vals no callback set 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe3867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe3867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.953+0000 7fe3867fc640 10 monclient: dump: 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:52.960 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:52.961 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 10 monclient: shutdown 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 1 -- 172.21.15.26:0/2438120597 >> v1:172.21.15.26:6790/0 conn(0x7fe3880b1730 legacy=0x7fe3880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 1 -- 172.21.15.26:0/2438120597 shutdown_connections 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 1 -- 172.21.15.26:0/2438120597 >> 172.21.15.26:0/2438120597 conn(0x7fe3880617a0 msgr2=0x7fe388061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 1 -- 172.21.15.26:0/2438120597 shutdown_connections 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.954+0000 7fe38e154640 1 -- 172.21.15.26:0/2438120597 wait complete. 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.955+0000 7fe38e154640 10 monclient: build_initial_monmap 2024-03-25T14:33:52.962 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.955+0000 7fe38e154640 10 monclient: monmap: 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 1 librados: starting msgr at 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 1 librados: starting objecter 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 1 Processor -- start 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 1 -- start start 2024-03-25T14:33:52.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 1 librados: setting wanted keys 2024-03-25T14:33:52.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 1 librados: calling monclient init 2024-03-25T14:33:52.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.956+0000 7fe38e154640 10 monclient: init 2024-03-25T14:33:52.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient: _reopen_session rank -1 2024-03-25T14:33:52.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:33:52.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient(hunting): picked mon.noname-c con 0x7fe3880b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient(hunting): picked mon.noname-b con 0x7fe3880b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient(hunting): picked mon.noname-a con 0x7fe38814be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3881dafb0 con 0x7fe38814be90 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3881432d0 con 0x7fe3880b4d60 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3881dbbe0 con 0x7fe3880b1730 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient(hunting): _renew_subs 2024-03-25T14:33:52.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.957+0000 7fe38e154640 10 monclient(hunting): authenticate will time out at 1918.060946s 2024-03-25T14:33:52.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.961+0000 7fe3877fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe3880b1730 0x7fe3881db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57480/0 (socket says 172.21.15.26:57480) 2024-03-25T14:33:52.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.961+0000 7fe3877fe640 1 -- 172.21.15.26:0/1482791376 learned_addr learned my addr 172.21.15.26:0/1482791376 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:33:52.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.962+0000 7fe36ffff640 10 client.?.objecter ms_handle_connect 0x7fe3880b1730 2024-03-25T14:33:52.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.963+0000 7fe36ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.974+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062192351 0 0) 0x7fe3881dbbe0 con 0x7fe3880b1730 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.974+0000 7fe36ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.974+0000 7fe36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.974+0000 7fe36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.974+0000 7fe36ffff640 10 monclient(hunting): my global_id is 4487 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.974+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe374004eb0 con 0x7fe3880b1730 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.975+0000 7fe36ffff640 10 client.?.objecter ms_handle_connect 0x7fe3880b4d60 2024-03-25T14:33:52.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.975+0000 7fe36ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:52.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.984+0000 7fe36ffff640 10 client.?.objecter ms_handle_connect 0x7fe38814be90 2024-03-25T14:33:52.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.984+0000 7fe36ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:33:52.999 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.993+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697430175 0 0) 0x7fe3881432d0 con 0x7fe3880b4d60 2024-03-25T14:33:53.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.993+0000 7fe36ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:53.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.993+0000 7fe36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:53.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.993+0000 7fe36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:53.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.993+0000 7fe36ffff640 10 monclient(hunting): my global_id is 4522 2024-03-25T14:33:53.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.993+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3881dbbe0 con 0x7fe3880b4d60 2024-03-25T14:33:53.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183971326 0 0) 0x7fe374004eb0 con 0x7fe3880b1730 2024-03-25T14:33:53.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3881432d0 con 0x7fe3880b1730 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe378001a10 con 0x7fe3880b1730 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 10 monclient(hunting): dump: 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:53.002 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:53.003 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:53.003 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:53.003 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:53.003 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:53.003 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:53.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:52.995+0000 7fe36ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:33:53.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.000+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 79257900 0 0) 0x7fe3881dbbe0 con 0x7fe3880b4d60 2024-03-25T14:33:53.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.000+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe374004eb0 con 0x7fe3880b4d60 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.000+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe37c001b00 con 0x7fe3880b4d60 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.000+0000 7fe36ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.000+0000 7fe36ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.000+0000 7fe36ffff640 10 monclient(hunting): dump: 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:53.007 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:53.008 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:53.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.004+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118445124 0 0) 0x7fe3881dafb0 con 0x7fe38814be90 2024-03-25T14:33:53.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.004+0000 7fe36ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:33:53.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.004+0000 7fe36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:33:53.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.004+0000 7fe36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:33:53.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.004+0000 7fe36ffff640 10 monclient(hunting): my global_id is 4536 2024-03-25T14:33:53.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.004+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3881dbbe0 con 0x7fe38814be90 2024-03-25T14:33:53.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.008+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2335181766 0 0) 0x7fe374004eb0 con 0x7fe3880b4d60 2024-03-25T14:33:53.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.008+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 >> v1:172.21.15.26:6790/0 conn(0x7fe3880b1730 legacy=0x7fe3881db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:53.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.008+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 >> v1:172.21.15.26:6789/0 conn(0x7fe38814be90 legacy=0x7fe38813b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:53.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.008+0000 7fe36ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.008+0000 7fe36ffff640 1 monclient: found mon.b 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe36ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe36ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388139670 con 0x7fe3880b4d60 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe36ffff640 10 monclient: _finish_auth 0 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe36ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe36ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 5 monclient: authenticate success, global_id 4522 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 10 monclient: _renew_subs 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38813bbe0 con 0x7fe3880b4d60 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:33:53.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 10 monclient: _renew_subs 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe388142a30 con 0x7fe3880b4d60 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.009+0000 7fe38e154640 1 librados: init done 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.010+0000 7fe38e154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.010+0000 7fe38e154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.010+0000 7fe38e154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:53.017 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.010+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3881db140 con 0x7fe3880b4d60 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe37c004a90 con 0x7fe3880b4d60 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe37c001cc0 con 0x7fe3880b4d60 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 10 monclient: dump: 2024-03-25T14:33:53.311 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:33:53.312 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe385ffb640 4 set_mon_vals no callback set 2024-03-25T14:33:53.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.305+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fe37c01ac10 con 0x7fe3880b4d60 2024-03-25T14:33:53.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.306+0000 7fe3877fe640 10 monclient: get_auth_request con 0x7fe37406df30 auth_method 0 2024-03-25T14:33:53.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.306+0000 7fe36ffff640 10 client.4522.objecter ms_handle_connect 0x7fe37406df30 2024-03-25T14:33:53.317 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.311+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fe37c054810 con 0x7fe3880b4d60 2024-03-25T14:33:53.317 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.311+0000 7fe36ffff640 10 client.4522.objecter ms_dispatch 0x7fe388000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:33:53.317 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.311+0000 7fe36ffff640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:33:53.317 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.311+0000 7fe36ffff640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:33:53.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.311+0000 7fe36ffff640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T14:33:53.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.312+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe37c050320 con 0x7fe3880b4d60 2024-03-25T14:33:53.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.312+0000 7fe36ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:33:53.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.312+0000 7fe36ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:33:53.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.524+0000 7fe38e154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:53.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.524+0000 7fe38e154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:53.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.524+0000 7fe38e154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:33:53.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.524+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3880627b0 con 0x7fe3880b4d60 2024-03-25T14:33:53.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.570+0000 7fe36ffff640 1 -- 172.21.15.26:0/1482791376 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fe37c052250 con 0x7fe3880b4d60 2024-03-25T14:33:53.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.570+0000 7fe36ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:33:53.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.570+0000 7fe36ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:33:53.577 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:33:53.577 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 10 librados: watch_flush enter 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 10 librados: watch_flush exit 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 10 client.4522.objecter successfully canceled tick 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 >> v1:172.21.15.121:6800/288813066 conn(0x7fe37406df30 legacy=0x7fe3740703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 10 monclient: shutdown 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 >> v1:172.21.15.121:6789/0 conn(0x7fe3880b4d60 legacy=0x7fe388138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:33:53.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 shutdown_connections 2024-03-25T14:33:53.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 >> 172.21.15.26:0/1482791376 conn(0x7fe3880617a0 msgr2=0x7fe388137300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:33:53.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 shutdown_connections 2024-03-25T14:33:53.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 -- 172.21.15.26:0/1482791376 wait complete. 2024-03-25T14:33:53.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:33:53.573+0000 7fe38e154640 1 librados: shutdown 2024-03-25T14:33:53.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:00.399 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:00.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.511+0000 7ff76429d640 10 monclient: get_monmap_and_config 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.511+0000 7ff76429d640 10 monclient: build_initial_monmap 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.511+0000 7ff76429d640 10 monclient: monmap: 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:00.512508+0000 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:00.512508+0000 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:00.517 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.512+0000 7ff76429d640 1 Processor -- start 2024-03-25T14:34:00.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.512+0000 7ff76429d640 1 -- start start 2024-03-25T14:34:00.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.512+0000 7ff76429d640 10 monclient: init 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient(hunting): picked mon.noname-a con 0x7ff75c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient(hunting): picked mon.noname-b con 0x7ff75c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient(hunting): picked mon.noname-c con 0x7ff75c1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c142520 con 0x7ff75c0a98f0 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c0b28a0 con 0x7ff75c1430c0 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c0b2a30 con 0x7ff75c13f710 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:00.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.513+0000 7ff76429d640 10 monclient(hunting): authenticate will time out at 1925.616355s 2024-03-25T14:34:00.523 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.518+0000 7ff762039640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff75c0a98f0 0x7ff75c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45042/0 (socket says 172.21.15.26:45042) 2024-03-25T14:34:00.523 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.518+0000 7ff762039640 1 -- 172.21.15.26:0/4254138340 learned_addr learned my addr 172.21.15.26:0/4254138340 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:00.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.530+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524737403 0 0) 0x7ff75c0b2a30 con 0x7ff75c13f710 2024-03-25T14:34:00.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.530+0000 7ff761037640 10 monclient(hunting): _negotiate 2024-03-25T14:34:00.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.530+0000 7ff761037640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:00.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.530+0000 7ff761037640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:00.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.530+0000 7ff761037640 10 monclient(hunting): my global_id is 4490 2024-03-25T14:34:00.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.530+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff748003520 con 0x7ff75c13f710 2024-03-25T14:34:00.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.538+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311474845 0 0) 0x7ff75c0b28a0 con 0x7ff75c1430c0 2024-03-25T14:34:00.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.538+0000 7ff761037640 10 monclient(hunting): _negotiate 2024-03-25T14:34:00.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.538+0000 7ff761037640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:00.543 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.538+0000 7ff761037640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:00.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.538+0000 7ff761037640 10 monclient(hunting): my global_id is 4525 2024-03-25T14:34:00.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.538+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff75c0b2a30 con 0x7ff75c1430c0 2024-03-25T14:34:00.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.540+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895933490 0 0) 0x7ff75c142520 con 0x7ff75c0a98f0 2024-03-25T14:34:00.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.540+0000 7ff761037640 10 monclient(hunting): _negotiate 2024-03-25T14:34:00.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.540+0000 7ff761037640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:00.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.540+0000 7ff761037640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:00.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.540+0000 7ff761037640 10 monclient(hunting): my global_id is 4539 2024-03-25T14:34:00.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.540+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff75c0b28a0 con 0x7ff75c0a98f0 2024-03-25T14:34:00.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3489123759 0 0) 0x7ff748003520 con 0x7ff75c13f710 2024-03-25T14:34:00.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 >> v1:172.21.15.121:6789/0 conn(0x7ff75c1430c0 legacy=0x7ff75c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:00.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 >> v1:172.21.15.26:6789/0 conn(0x7ff75c0a98f0 legacy=0x7ff75c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:00.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: _finish_hunting 0 2024-03-25T14:34:00.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 1 monclient: found mon.noname-b 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c0b44c0 con 0x7ff75c13f710 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: _finish_auth 0 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: _check_auth_tickets 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff74c001690 con 0x7ff75c13f710 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff761037640 10 monclient: dump: 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.549 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff76429d640 5 monclient: authenticate success, global_id 4490 2024-03-25T14:34:00.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.543+0000 7ff76429d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:00.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff74c0017f0 con 0x7ff75c13f710 2024-03-25T14:34:00.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff761037640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:00.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff761037640 1 -- 172.21.15.26:0/4254138340 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff74c001c00 con 0x7ff75c13f710 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff76429d640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff76429d640 4 set_mon_vals no callback set 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff761037640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff761037640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.557+0000 7ff761037640 10 monclient: dump: 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:00.563 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:00.564 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:00.564 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 10 monclient: shutdown 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 1 -- 172.21.15.26:0/4254138340 >> v1:172.21.15.26:6790/0 conn(0x7ff75c13f710 legacy=0x7ff75c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 1 -- 172.21.15.26:0/4254138340 shutdown_connections 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 1 -- 172.21.15.26:0/4254138340 >> 172.21.15.26:0/4254138340 conn(0x7ff75c0b7ea0 msgr2=0x7ff75c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 1 -- 172.21.15.26:0/4254138340 shutdown_connections 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.558+0000 7ff76429d640 1 -- 172.21.15.26:0/4254138340 wait complete. 2024-03-25T14:34:00.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.559+0000 7ff76429d640 10 monclient: build_initial_monmap 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.559+0000 7ff76429d640 10 monclient: monmap: 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 1 librados: starting msgr at 2024-03-25T14:34:00.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 1 librados: starting objecter 2024-03-25T14:34:00.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 1 Processor -- start 2024-03-25T14:34:00.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 1 -- start start 2024-03-25T14:34:00.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 1 librados: setting wanted keys 2024-03-25T14:34:00.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 1 librados: calling monclient init 2024-03-25T14:34:00.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.560+0000 7ff76429d640 10 monclient: init 2024-03-25T14:34:00.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:00.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:00.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient(hunting): picked mon.noname-a con 0x7ff75c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:00.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient(hunting): picked mon.noname-b con 0x7ff75c13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:00.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient(hunting): picked mon.noname-c con 0x7ff75c1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:00.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c0aba00 con 0x7ff75c0a98f0 2024-03-25T14:34:00.569 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c1d7490 con 0x7ff75c13f710 2024-03-25T14:34:00.569 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c1d8360 con 0x7ff75c1430c0 2024-03-25T14:34:00.569 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:00.569 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.561+0000 7ff76429d640 10 monclient(hunting): authenticate will time out at 1925.663903s 2024-03-25T14:34:00.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.568+0000 7ff76283a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff75c1430c0 0x7ff75c133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47124/0 (socket says 172.21.15.26:47124) 2024-03-25T14:34:00.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.568+0000 7ff76283a640 1 -- 172.21.15.26:0/3846417763 learned_addr learned my addr 172.21.15.26:0/3846417763 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:00.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.574+0000 7ff752ffd640 10 client.?.objecter ms_handle_connect 0x7ff75c1430c0 2024-03-25T14:34:00.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.574+0000 7ff752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:00.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.574+0000 7ff752ffd640 10 client.?.objecter ms_handle_connect 0x7ff75c13f710 2024-03-25T14:34:00.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.574+0000 7ff752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:00.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.577+0000 7ff752ffd640 10 client.?.objecter ms_handle_connect 0x7ff75c0a98f0 2024-03-25T14:34:00.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.577+0000 7ff752ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:00.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.580+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070896525 0 0) 0x7ff75c1d8360 con 0x7ff75c1430c0 2024-03-25T14:34:00.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.580+0000 7ff752ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:00.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.580+0000 7ff752ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:00.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.580+0000 7ff752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:00.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.580+0000 7ff752ffd640 10 monclient(hunting): my global_id is 4493 2024-03-25T14:34:00.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.580+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff740004eb0 con 0x7ff75c1430c0 2024-03-25T14:34:00.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.585+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948469047 0 0) 0x7ff75c1d7490 con 0x7ff75c13f710 2024-03-25T14:34:00.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.585+0000 7ff752ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:00.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.585+0000 7ff752ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:00.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.585+0000 7ff752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:00.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.585+0000 7ff752ffd640 10 monclient(hunting): my global_id is 4528 2024-03-25T14:34:00.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.585+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff75c1d8360 con 0x7ff75c13f710 2024-03-25T14:34:00.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.588+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752512113 0 0) 0x7ff75c0aba00 con 0x7ff75c0a98f0 2024-03-25T14:34:00.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.588+0000 7ff752ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:00.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.588+0000 7ff752ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:00.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.588+0000 7ff752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:00.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.588+0000 7ff752ffd640 10 monclient(hunting): my global_id is 4542 2024-03-25T14:34:00.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.588+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff75c1d7490 con 0x7ff75c0a98f0 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358995231 0 0) 0x7ff75c1d8360 con 0x7ff75c13f710 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff75c0aba00 con 0x7ff75c13f710 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff74c001b00 con 0x7ff75c13f710 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 10 monclient(hunting): dump: 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:00.599 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.594+0000 7ff752ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:00.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.601+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 689350927 0 0) 0x7ff740004eb0 con 0x7ff75c1430c0 2024-03-25T14:34:00.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.601+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff75c1d8360 con 0x7ff75c1430c0 2024-03-25T14:34:00.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.601+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff754001c90 con 0x7ff75c1430c0 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.601+0000 7ff752ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.601+0000 7ff752ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.601+0000 7ff752ffd640 10 monclient(hunting): dump: 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:00.607 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1534389516 0 0) 0x7ff75c0aba00 con 0x7ff75c13f710 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 >> v1:172.21.15.26:6790/0 conn(0x7ff75c1430c0 legacy=0x7ff75c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 >> v1:172.21.15.26:6789/0 conn(0x7ff75c0a98f0 legacy=0x7ff75c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 1 monclient: found mon.b 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:00.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c0abc30 con 0x7ff75c13f710 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 10 monclient: _finish_auth 0 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff752ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff76429d640 5 monclient: authenticate success, global_id 4528 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff76429d640 10 monclient: _renew_subs 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff76429d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.612+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c137940 con 0x7ff75c13f710 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.613+0000 7ff76429d640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.613+0000 7ff76429d640 10 monclient: _renew_subs 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.613+0000 7ff76429d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.613+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c135220 con 0x7ff75c13f710 2024-03-25T14:34:00.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.613+0000 7ff76429d640 1 librados: init done 2024-03-25T14:34:00.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.614+0000 7ff76429d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:00.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.614+0000 7ff76429d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:00.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.614+0000 7ff76429d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:00.620 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.614+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75c0627b0 con 0x7ff75c13f710 2024-03-25T14:34:00.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff74c0038a0 con 0x7ff75c13f710 2024-03-25T14:34:00.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff752ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:00.892 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff74c001cc0 con 0x7ff75c13f710 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff752ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff752ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff752ffd640 10 monclient: dump: 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:00.893 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:00.894 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:00.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.887+0000 7ff760836640 4 set_mon_vals no callback set 2024-03-25T14:34:00.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.888+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ff74c019a20 con 0x7ff75c13f710 2024-03-25T14:34:00.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.889+0000 7ff762039640 10 monclient: get_auth_request con 0x7ff74006df70 auth_method 0 2024-03-25T14:34:00.894 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.889+0000 7ff752ffd640 10 client.4528.objecter ms_handle_connect 0x7ff74006df70 2024-03-25T14:34:00.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.891+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7ff74c0535a0 con 0x7ff75c13f710 2024-03-25T14:34:00.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.891+0000 7ff752ffd640 10 client.4528.objecter ms_dispatch 0x7ff75c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:00.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.891+0000 7ff752ffd640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:00.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.891+0000 7ff752ffd640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:00.896 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.891+0000 7ff752ffd640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T14:34:00.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.898+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff74c04f0b0 con 0x7ff75c13f710 2024-03-25T14:34:00.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.898+0000 7ff752ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:00.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:00.898+0000 7ff752ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:01.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.159+0000 7ff76429d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:01.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.159+0000 7ff76429d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:01.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.159+0000 7ff76429d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:01.165 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.159+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff75c069f40 con 0x7ff75c13f710 2024-03-25T14:34:01.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.185+0000 7ff752ffd640 1 -- 172.21.15.26:0/3846417763 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7ff74c050fe0 con 0x7ff75c13f710 2024-03-25T14:34:01.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.185+0000 7ff752ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:01.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.185+0000 7ff752ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:01.190 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:01.190 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 10 librados: watch_flush enter 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 10 librados: watch_flush exit 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 10 client.4528.objecter successfully canceled tick 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 >> v1:172.21.15.121:6800/288813066 conn(0x7ff74006df70 legacy=0x7ff740070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 10 monclient: shutdown 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:01.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 >> v1:172.21.15.121:6789/0 conn(0x7ff75c13f710 legacy=0x7ff75c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:01.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 shutdown_connections 2024-03-25T14:34:01.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.187+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 >> 172.21.15.26:0/3846417763 conn(0x7ff75c0b7ea0 msgr2=0x7ff75c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:01.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.188+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 shutdown_connections 2024-03-25T14:34:01.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.188+0000 7ff76429d640 1 -- 172.21.15.26:0/3846417763 wait complete. 2024-03-25T14:34:01.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:01.188+0000 7ff76429d640 1 librados: shutdown 2024-03-25T14:34:01.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:02.022 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:20.882 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.031 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:20.892 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.070 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:20.930 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.085 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:20.961 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.089 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:20.950 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.092 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:20.969 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.094 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:20.954 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.120 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:20.996 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:02.132 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:20.992 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:08.013 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.124+0000 7f8e7714b640 10 monclient: get_monmap_and_config 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.124+0000 7f8e7714b640 10 monclient: build_initial_monmap 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.124+0000 7f8e7714b640 10 monclient: monmap: 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:08.125122+0000 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:08.125122+0000 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:08.129 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:08.130 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:08.130 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:08.130 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:08.130 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.125+0000 7f8e7714b640 1 Processor -- start 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.125+0000 7f8e7714b640 1 -- start start 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.125+0000 7f8e7714b640 10 monclient: init 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e700b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:08.131 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e700b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:08.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e7014be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:08.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e700af940 con 0x7f8e700b4d60 2024-03-25T14:34:08.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e700b0a70 con 0x7f8e700b1730 2024-03-25T14:34:08.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e700b0d50 con 0x7f8e7014be90 2024-03-25T14:34:08.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:08.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.126+0000 7f8e7714b640 10 monclient(hunting): authenticate will time out at 1933.228959s 2024-03-25T14:34:08.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.137+0000 7f8e74ee7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f8e700b4d60 0x7f8e7013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41006/0 (socket says 172.21.15.26:41006) 2024-03-25T14:34:08.141 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.137+0000 7f8e74ee7640 1 -- 172.21.15.26:0/3966366586 learned_addr learned my addr 172.21.15.26:0/3966366586 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:08.155 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.150+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180917057 0 0) 0x7f8e700b0d50 con 0x7f8e7014be90 2024-03-25T14:34:08.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.150+0000 7f8e6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:08.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.150+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:08.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.150+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:08.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.150+0000 7f8e6f7fe640 10 monclient(hunting): my global_id is 4496 2024-03-25T14:34:08.156 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.150+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e60003520 con 0x7f8e7014be90 2024-03-25T14:34:08.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.157+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372104781 0 0) 0x7f8e700b0a70 con 0x7f8e700b1730 2024-03-25T14:34:08.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.157+0000 7f8e6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:08.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.157+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:08.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.157+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:08.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.157+0000 7f8e6f7fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T14:34:08.162 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.157+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e700b0d50 con 0x7f8e700b1730 2024-03-25T14:34:08.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.158+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350304659 0 0) 0x7f8e700af940 con 0x7f8e700b4d60 2024-03-25T14:34:08.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.158+0000 7f8e6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:08.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.158+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:08.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.158+0000 7f8e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:08.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.158+0000 7f8e6f7fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T14:34:08.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.158+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e700b0a70 con 0x7f8e700b4d60 2024-03-25T14:34:08.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474702538 0 0) 0x7f8e700b0d50 con 0x7f8e700b1730 2024-03-25T14:34:08.177 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 >> v1:172.21.15.26:6790/0 conn(0x7f8e7014be90 legacy=0x7f8e7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 >> v1:172.21.15.26:6789/0 conn(0x7f8e700b4d60 legacy=0x7f8e7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 1 monclient: found mon.noname-c 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e700b1030 con 0x7f8e700b1730 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8e5c001670 con 0x7f8e700b1730 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:34:08.178 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e6f7fe640 10 monclient: dump: 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e7714b640 5 monclient: authenticate success, global_id 4531 2024-03-25T14:34:08.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.172+0000 7f8e7714b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e5c0017d0 con 0x7f8e700b1730 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e6f7fe640 1 -- 172.21.15.26:0/3966366586 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8e5c001be0 con 0x7f8e700b1730 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 4 set_mon_vals no callback set 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:08.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e6f7fe640 10 monclient: dump: 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:08.198 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 10 monclient: shutdown 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 1 -- 172.21.15.26:0/3966366586 >> v1:172.21.15.121:6789/0 conn(0x7f8e700b1730 legacy=0x7f8e700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 1 -- 172.21.15.26:0/3966366586 shutdown_connections 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.192+0000 7f8e7714b640 1 -- 172.21.15.26:0/3966366586 >> 172.21.15.26:0/3966366586 conn(0x7f8e700617a0 msgr2=0x7f8e70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.193+0000 7f8e7714b640 1 -- 172.21.15.26:0/3966366586 shutdown_connections 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.193+0000 7f8e7714b640 1 -- 172.21.15.26:0/3966366586 wait complete. 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.193+0000 7f8e7714b640 10 monclient: build_initial_monmap 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.193+0000 7f8e7714b640 10 monclient: monmap: 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:08.200 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 1 librados: starting msgr at 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 1 librados: starting objecter 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 1 Processor -- start 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 1 -- start start 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 1 librados: setting wanted keys 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 1 librados: calling monclient init 2024-03-25T14:34:08.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.194+0000 7f8e7714b640 10 monclient: init 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e700b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e700b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7014be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e701447e0 con 0x7f8e700b1730 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e70143900 con 0x7f8e700b4d60 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e70143a90 con 0x7f8e7014be90 2024-03-25T14:34:08.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:08.203 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.195+0000 7f8e7714b640 10 monclient(hunting): authenticate will time out at 1933.298293s 2024-03-25T14:34:08.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.210+0000 7f8e6ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8e700b4d60 0x7f8e701dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:35318/0 (socket says 172.21.15.26:35318) 2024-03-25T14:34:08.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.210+0000 7f8e6ffff640 1 -- 172.21.15.26:0/2438945887 learned_addr learned my addr 172.21.15.26:0/2438945887 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:08.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.211+0000 7f8e6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8e700b4d60 2024-03-25T14:34:08.215 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.211+0000 7f8e6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:08.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.214+0000 7f8e6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8e700b1730 2024-03-25T14:34:08.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.214+0000 7f8e6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:08.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.215+0000 7f8e6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8e7014be90 2024-03-25T14:34:08.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.215+0000 7f8e6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:08.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.219+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067892438 0 0) 0x7f8e70143900 con 0x7f8e700b4d60 2024-03-25T14:34:08.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.219+0000 7f8e6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:08.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.219+0000 7f8e6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:08.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.219+0000 7f8e6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:08.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.219+0000 7f8e6d7fa640 10 monclient(hunting): my global_id is 4534 2024-03-25T14:34:08.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.219+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e54004eb0 con 0x7f8e700b4d60 2024-03-25T14:34:08.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.221+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321318765 0 0) 0x7f8e70143a90 con 0x7f8e7014be90 2024-03-25T14:34:08.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.221+0000 7f8e6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:08.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.221+0000 7f8e6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:08.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.221+0000 7f8e6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:08.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.221+0000 7f8e6d7fa640 10 monclient(hunting): my global_id is 4499 2024-03-25T14:34:08.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.221+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e70143900 con 0x7f8e7014be90 2024-03-25T14:34:08.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.228+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209679253 0 0) 0x7f8e701447e0 con 0x7f8e700b1730 2024-03-25T14:34:08.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.228+0000 7f8e6d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:08.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.228+0000 7f8e6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:08.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.228+0000 7f8e6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:08.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.228+0000 7f8e6d7fa640 10 monclient(hunting): my global_id is 4548 2024-03-25T14:34:08.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.228+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e70143a90 con 0x7f8e700b1730 2024-03-25T14:34:08.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3365944364 0 0) 0x7f8e54004eb0 con 0x7f8e700b4d60 2024-03-25T14:34:08.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e701447e0 con 0x7f8e700b4d60 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8e5c001670 con 0x7f8e700b4d60 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 10 monclient(hunting): dump: 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:08.250 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:08.251 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.244+0000 7f8e6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.248+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549224738 0 0) 0x7f8e70143900 con 0x7f8e7014be90 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.248+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e54004eb0 con 0x7f8e7014be90 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.248+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8e68001a70 con 0x7f8e7014be90 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.248+0000 7f8e6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.248+0000 7f8e6d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.248+0000 7f8e6d7fa640 10 monclient(hunting): dump: 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:08.253 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:08.254 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1193036290 0 0) 0x7f8e701447e0 con 0x7f8e700b4d60 2024-03-25T14:34:08.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 >> v1:172.21.15.26:6790/0 conn(0x7f8e7014be90 legacy=0x7f8e701db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 >> v1:172.21.15.26:6789/0 conn(0x7f8e700b1730 legacy=0x7f8e7012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 1 monclient: found mon.b 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70130600 con 0x7f8e700b4d60 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 5 monclient: authenticate success, global_id 4534 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 10 monclient: _renew_subs 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:08.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e70144170 con 0x7f8e700b4d60 2024-03-25T14:34:08.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:08.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 10 monclient: _renew_subs 2024-03-25T14:34:08.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:08.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7013ea40 con 0x7f8e700b4d60 2024-03-25T14:34:08.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.253+0000 7f8e7714b640 1 librados: init done 2024-03-25T14:34:08.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.254+0000 7f8e7714b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:08.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.254+0000 7f8e7714b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:08.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.254+0000 7f8e7714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:08.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.254+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e700627b0 con 0x7f8e700b4d60 2024-03-25T14:34:08.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e5c0038a0 con 0x7f8e700b4d60 2024-03-25T14:34:08.551 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8e5c003cb0 con 0x7f8e700b4d60 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6d7fa640 10 monclient: dump: 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:08.552 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:08.553 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:08.553 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:08.553 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:08.553 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:08.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.546+0000 7f8e6effd640 4 set_mon_vals no callback set 2024-03-25T14:34:08.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.547+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f8e5c003ed0 con 0x7f8e700b4d60 2024-03-25T14:34:08.554 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.548+0000 7f8e74ee7640 10 monclient: get_auth_request con 0x7f8e5406df70 auth_method 0 2024-03-25T14:34:08.554 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.548+0000 7f8e6d7fa640 10 client.4534.objecter ms_handle_connect 0x7f8e5406df70 2024-03-25T14:34:08.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f8e5c054920 con 0x7f8e700b4d60 2024-03-25T14:34:08.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 10 client.4534.objecter ms_dispatch 0x7f8e70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:08.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:08.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:08.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T14:34:08.558 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e5c050430 con 0x7f8e700b4d60 2024-03-25T14:34:08.558 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:08.558 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.552+0000 7f8e6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:08.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.808+0000 7f8e7714b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:08.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.808+0000 7f8e7714b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:08.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.808+0000 7f8e7714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:08.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.808+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e7013ef10 con 0x7f8e700b4d60 2024-03-25T14:34:08.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.839+0000 7f8e6d7fa640 1 -- 172.21.15.26:0/2438945887 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f8e5c052360 con 0x7f8e700b4d60 2024-03-25T14:34:08.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.839+0000 7f8e6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:08.844 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.839+0000 7f8e6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:08.845 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:08.845 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:08.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 10 librados: watch_flush enter 2024-03-25T14:34:08.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 10 librados: watch_flush exit 2024-03-25T14:34:08.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T14:34:08.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 10 client.4534.objecter successfully canceled tick 2024-03-25T14:34:08.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 >> v1:172.21.15.121:6800/288813066 conn(0x7f8e5406df70 legacy=0x7f8e54070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 10 monclient: shutdown 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.841+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 >> v1:172.21.15.121:6789/0 conn(0x7f8e700b4d60 legacy=0x7f8e701dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.842+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 shutdown_connections 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.842+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 >> 172.21.15.26:0/2438945887 conn(0x7f8e700617a0 msgr2=0x7f8e700b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.842+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 shutdown_connections 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.842+0000 7f8e7714b640 1 -- 172.21.15.26:0/2438945887 wait complete. 2024-03-25T14:34:08.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:08.842+0000 7f8e7714b640 1 librados: shutdown 2024-03-25T14:34:08.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:15.667 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:15.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.775+0000 7fe164d44640 10 monclient: get_monmap_and_config 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.775+0000 7fe164d44640 10 monclient: build_initial_monmap 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.775+0000 7fe164d44640 10 monclient: monmap: 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:15.776634+0000 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:15.776634+0000 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:15.781 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.776+0000 7fe164d44640 1 Processor -- start 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.776+0000 7fe164d44640 1 -- start start 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.776+0000 7fe164d44640 10 monclient: init 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient(hunting): picked mon.noname-a con 0x7fe1600a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient(hunting): picked mon.noname-c con 0x7fe16013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient(hunting): picked mon.noname-b con 0x7fe1601430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160142520 con 0x7fe1600a98f0 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600b28a0 con 0x7fe16013f710 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600b2a30 con 0x7fe1601430c0 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:15.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.777+0000 7fe164d44640 10 monclient(hunting): authenticate will time out at 1940.880465s 2024-03-25T14:34:15.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.791+0000 7fe15dd9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe16013f710 0x7fe1601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39670/0 (socket says 172.21.15.26:39670) 2024-03-25T14:34:15.795 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.791+0000 7fe15dd9b640 1 -- 172.21.15.26:0/235153833 learned_addr learned my addr 172.21.15.26:0/235153833 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:15.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.803+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772749735 0 0) 0x7fe1600b28a0 con 0x7fe16013f710 2024-03-25T14:34:15.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.803+0000 7fe15d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:34:15.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.803+0000 7fe15d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:15.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.803+0000 7fe15d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:15.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.803+0000 7fe15d59a640 10 monclient(hunting): my global_id is 4537 2024-03-25T14:34:15.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.803+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148003520 con 0x7fe16013f710 2024-03-25T14:34:15.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.808+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312628356 0 0) 0x7fe160142520 con 0x7fe1600a98f0 2024-03-25T14:34:15.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.808+0000 7fe15d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:34:15.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.808+0000 7fe15d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:15.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.808+0000 7fe15d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:15.813 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.808+0000 7fe15d59a640 10 monclient(hunting): my global_id is 4551 2024-03-25T14:34:15.814 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.808+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1600b28a0 con 0x7fe1600a98f0 2024-03-25T14:34:15.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.813+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509541289 0 0) 0x7fe1600b2a30 con 0x7fe1601430c0 2024-03-25T14:34:15.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.813+0000 7fe15d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:34:15.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.813+0000 7fe15d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:15.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.813+0000 7fe15d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:15.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.813+0000 7fe15d59a640 10 monclient(hunting): my global_id is 4502 2024-03-25T14:34:15.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.813+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160142520 con 0x7fe1601430c0 2024-03-25T14:34:15.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840121125 0 0) 0x7fe148003520 con 0x7fe16013f710 2024-03-25T14:34:15.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 >> v1:172.21.15.26:6790/0 conn(0x7fe1601430c0 legacy=0x7fe16014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:15.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 >> v1:172.21.15.26:6789/0 conn(0x7fe1600a98f0 legacy=0x7fe1600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:15.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: _finish_hunting 0 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 1 monclient: found mon.noname-c 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600b44c0 con 0x7fe16013f710 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: _finish_auth 0 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: _check_auth_tickets 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe154001690 con 0x7fe16013f710 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe15d59a640 10 monclient: dump: 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.821 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe164d44640 5 monclient: authenticate success, global_id 4537 2024-03-25T14:34:15.822 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.815+0000 7fe164d44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:15.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1540017f0 con 0x7fe16013f710 2024-03-25T14:34:15.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe15d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:15.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe15d59a640 1 -- 172.21.15.26:0/235153833 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe154001c00 con 0x7fe16013f710 2024-03-25T14:34:15.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe15d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe15d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe15d59a640 10 monclient: dump: 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.834 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe164d44640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:15.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.828+0000 7fe164d44640 4 set_mon_vals no callback set 2024-03-25T14:34:15.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 10 monclient: shutdown 2024-03-25T14:34:15.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 1 -- 172.21.15.26:0/235153833 >> v1:172.21.15.121:6789/0 conn(0x7fe16013f710 legacy=0x7fe1601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 1 -- 172.21.15.26:0/235153833 shutdown_connections 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 1 -- 172.21.15.26:0/235153833 >> 172.21.15.26:0/235153833 conn(0x7fe1600b7ea0 msgr2=0x7fe1600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 1 -- 172.21.15.26:0/235153833 shutdown_connections 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.829+0000 7fe164d44640 1 -- 172.21.15.26:0/235153833 wait complete. 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 10 monclient: build_initial_monmap 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 10 monclient: monmap: 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:15.837 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 1 librados: starting msgr at 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 1 librados: starting objecter 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 1 Processor -- start 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 1 -- start start 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 1 librados: setting wanted keys 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 1 librados: calling monclient init 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.830+0000 7fe164d44640 10 monclient: init 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:34:15.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient(hunting): picked mon.noname-b con 0x7fe1600a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient(hunting): picked mon.noname-c con 0x7fe16013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient(hunting): picked mon.noname-a con 0x7fe1601430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160138c10 con 0x7fe1601430c0 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160138e60 con 0x7fe1600a98f0 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160139140 con 0x7fe16013f710 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:15.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.831+0000 7fe164d44640 10 monclient(hunting): authenticate will time out at 1940.934392s 2024-03-25T14:34:15.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.842+0000 7fe15ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe1601430c0 0x7fe1601e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45572/0 (socket says 172.21.15.26:45572) 2024-03-25T14:34:15.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.842+0000 7fe15ed9d640 1 -- 172.21.15.26:0/415127520 learned_addr learned my addr 172.21.15.26:0/415127520 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:15.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.845+0000 7fe14effd640 10 client.?.objecter ms_handle_connect 0x7fe1600a98f0 2024-03-25T14:34:15.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.846+0000 7fe14effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:15.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.849+0000 7fe14effd640 10 client.?.objecter ms_handle_connect 0x7fe1601430c0 2024-03-25T14:34:15.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.849+0000 7fe14effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:15.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.855+0000 7fe14effd640 10 client.?.objecter ms_handle_connect 0x7fe16013f710 2024-03-25T14:34:15.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.855+0000 7fe14effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:15.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.866+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906802106 0 0) 0x7fe160139140 con 0x7fe16013f710 2024-03-25T14:34:15.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.866+0000 7fe14effd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:15.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.866+0000 7fe14effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:15.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.866+0000 7fe14effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:15.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.866+0000 7fe14effd640 10 monclient(hunting): my global_id is 4505 2024-03-25T14:34:15.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.866+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe140004eb0 con 0x7fe16013f710 2024-03-25T14:34:15.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.867+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498652119 0 0) 0x7fe160138c10 con 0x7fe1601430c0 2024-03-25T14:34:15.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.867+0000 7fe14effd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:15.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.867+0000 7fe14effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:15.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.867+0000 7fe14effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:15.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.867+0000 7fe14effd640 10 monclient(hunting): my global_id is 4554 2024-03-25T14:34:15.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.867+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160139140 con 0x7fe1601430c0 2024-03-25T14:34:15.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.873+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912738420 0 0) 0x7fe160138e60 con 0x7fe1600a98f0 2024-03-25T14:34:15.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.873+0000 7fe14effd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:15.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.873+0000 7fe14effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:15.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.873+0000 7fe14effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:15.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.873+0000 7fe14effd640 10 monclient(hunting): my global_id is 4540 2024-03-25T14:34:15.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.873+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160138c10 con 0x7fe1600a98f0 2024-03-25T14:34:15.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 756173238 0 0) 0x7fe140004eb0 con 0x7fe16013f710 2024-03-25T14:34:15.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe160138e60 con 0x7fe16013f710 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe154001b00 con 0x7fe16013f710 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 10 monclient(hunting): dump: 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:15.883 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.877+0000 7fe14effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.884+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527869487 0 0) 0x7fe160138c10 con 0x7fe1600a98f0 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.884+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe140004eb0 con 0x7fe1600a98f0 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.884+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe150001930 con 0x7fe1600a98f0 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.884+0000 7fe14effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.884+0000 7fe14effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.884+0000 7fe14effd640 10 monclient(hunting): dump: 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:15.890 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.891 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.891 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:15.891 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:15.891 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:15.892 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:15.892 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:15.892 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.890+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1916071994 0 0) 0x7fe160139140 con 0x7fe1601430c0 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.890+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe160138c10 con 0x7fe1601430c0 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.890+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe158001c50 con 0x7fe1601430c0 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.890+0000 7fe14effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.890+0000 7fe14effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.890+0000 7fe14effd640 10 monclient(hunting): dump: 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:15.895 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:15.896 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:15.896 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:15.896 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:15.896 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:15.896 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:15.903 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 187144325 0 0) 0x7fe140004eb0 con 0x7fe1600a98f0 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 >> v1:172.21.15.26:6790/0 conn(0x7fe16013f710 legacy=0x7fe160132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 >> v1:172.21.15.26:6789/0 conn(0x7fe1601430c0 legacy=0x7fe1601e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 10 monclient: _finish_hunting 0 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 1 monclient: found mon.b 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160139920 con 0x7fe1600a98f0 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 10 monclient: _finish_auth 0 2024-03-25T14:34:15.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 10 monclient: _check_auth_tickets 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe14effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 5 monclient: authenticate success, global_id 4540 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 10 monclient: _renew_subs 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1601382e0 con 0x7fe1600a98f0 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 10 monclient: _renew_subs 2024-03-25T14:34:15.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:15.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1600ab8b0 con 0x7fe1600a98f0 2024-03-25T14:34:15.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.898+0000 7fe164d44640 1 librados: init done 2024-03-25T14:34:15.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.899+0000 7fe164d44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:15.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.899+0000 7fe164d44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:15.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.899+0000 7fe164d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:15.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:15.899+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe160137a20 con 0x7fe1600a98f0 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.190+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1500035f0 con 0x7fe1600a98f0 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.190+0000 7fe14effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.191+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe150003900 con 0x7fe1600a98f0 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.191+0000 7fe14effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.191+0000 7fe14effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.191+0000 7fe14effd640 10 monclient: dump: 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:16.196 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:16.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.191+0000 7fe15cd99640 4 set_mon_vals no callback set 2024-03-25T14:34:16.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.191+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fe1500198d0 con 0x7fe1600a98f0 2024-03-25T14:34:16.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.192+0000 7fe15dd9b640 10 monclient: get_auth_request con 0x7fe14006e160 auth_method 0 2024-03-25T14:34:16.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.192+0000 7fe14effd640 10 client.4540.objecter ms_handle_connect 0x7fe14006e160 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fe150053530 con 0x7fe1600a98f0 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 10 client.4540.objecter ms_dispatch 0x7fe160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe15004f040 con 0x7fe1600a98f0 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:16.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.196+0000 7fe14effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:16.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.420+0000 7fe164d44640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:16.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.420+0000 7fe164d44640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:16.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.420+0000 7fe164d44640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:16.426 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.420+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe16013d010 con 0x7fe1600a98f0 2024-03-25T14:34:16.449 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.445+0000 7fe14effd640 1 -- 172.21.15.26:0/415127520 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fe150050f70 con 0x7fe1600a98f0 2024-03-25T14:34:16.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.445+0000 7fe14effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:16.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.445+0000 7fe14effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:16.450 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:16.450 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:16.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 10 librados: watch_flush enter 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 10 librados: watch_flush exit 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 10 client.4540.objecter successfully canceled tick 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 >> v1:172.21.15.121:6800/288813066 conn(0x7fe14006e160 legacy=0x7fe140070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 10 monclient: shutdown 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 >> v1:172.21.15.121:6789/0 conn(0x7fe1600a98f0 legacy=0x7fe160137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 shutdown_connections 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 >> 172.21.15.26:0/415127520 conn(0x7fe1600b7ea0 msgr2=0x7fe160074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 shutdown_connections 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 -- 172.21.15.26:0/415127520 wait complete. 2024-03-25T14:34:16.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:16.447+0000 7fe164d44640 1 librados: shutdown 2024-03-25T14:34:16.465 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:22.074 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:40.934 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.083 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:40.944 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.120 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:07:40.981 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.128 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:40.989 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.141 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:41.002 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.153 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:41.029 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.161 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:41.037 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.172 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:07:41.033 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:22.179 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:07:41.055 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:23.272 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:23.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.386+0000 7f65d4d13640 10 monclient: get_monmap_and_config 2024-03-25T14:34:23.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.386+0000 7f65d4d13640 10 monclient: build_initial_monmap 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.386+0000 7f65d4d13640 10 monclient: monmap: 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:23.387517+0000 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:23.387517+0000 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:23.392 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.387+0000 7f65d4d13640 1 Processor -- start 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.387+0000 7f65d4d13640 1 -- start start 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.387+0000 7f65d4d13640 10 monclient: init 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient(hunting): picked mon.noname-c con 0x7f65d00a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient(hunting): picked mon.noname-a con 0x7f65d013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient(hunting): picked mon.noname-b con 0x7f65d01430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d0142520 con 0x7f65d013f710 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00b28a0 con 0x7f65d00a98f0 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00b2a30 con 0x7f65d01430c0 2024-03-25T14:34:23.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:23.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.388+0000 7f65d4d13640 10 monclient(hunting): authenticate will time out at 1948.491303s 2024-03-25T14:34:23.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.394+0000 7f65c7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f65d013f710 0x7f65d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59306/0 (socket says 172.21.15.26:59306) 2024-03-25T14:34:23.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.394+0000 7f65c7fff640 1 -- 172.21.15.26:0/2235065056 learned_addr learned my addr 172.21.15.26:0/2235065056 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:23.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.410+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089532807 0 0) 0x7f65d0142520 con 0x7f65d013f710 2024-03-25T14:34:23.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.410+0000 7f65cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:34:23.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.410+0000 7f65cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:23.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.410+0000 7f65cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:23.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.410+0000 7f65cdd9b640 10 monclient(hunting): my global_id is 4557 2024-03-25T14:34:23.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.410+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc003520 con 0x7f65d013f710 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999588460 0 0) 0x7f65d00b28a0 con 0x7f65d00a98f0 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): my global_id is 4543 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d0142520 con 0x7f65d00a98f0 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966494765 0 0) 0x7f65d00b2a30 con 0x7f65d01430c0 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:23.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:23.421 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 10 monclient(hunting): my global_id is 4508 2024-03-25T14:34:23.421 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.415+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d00b28a0 con 0x7f65d01430c0 2024-03-25T14:34:23.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1918973647 0 0) 0x7f65d0142520 con 0x7f65d00a98f0 2024-03-25T14:34:23.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 >> v1:172.21.15.26:6790/0 conn(0x7f65d01430c0 legacy=0x7f65d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:23.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 >> v1:172.21.15.26:6789/0 conn(0x7f65d013f710 legacy=0x7f65d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:23.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T14:34:23.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 1 monclient: found mon.noname-c 2024-03-25T14:34:23.428 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00b44c0 con 0x7f65d00a98f0 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: _finish_auth 0 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f65c0001670 con 0x7f65d00a98f0 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65cdd9b640 10 monclient: dump: 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.429 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65d4d13640 5 monclient: authenticate success, global_id 4543 2024-03-25T14:34:23.430 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.423+0000 7f65d4d13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:23.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65c00017d0 con 0x7f65d00a98f0 2024-03-25T14:34:23.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:23.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65cdd9b640 1 -- 172.21.15.26:0/2235065056 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f65c0001be0 con 0x7f65d00a98f0 2024-03-25T14:34:23.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65d4d13640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:23.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65d4d13640 4 set_mon_vals no callback set 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.436+0000 7f65cdd9b640 10 monclient: dump: 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:23.442 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:23.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 10 monclient: shutdown 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 1 -- 172.21.15.26:0/2235065056 >> v1:172.21.15.121:6789/0 conn(0x7f65d00a98f0 legacy=0x7f65d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 1 -- 172.21.15.26:0/2235065056 shutdown_connections 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 1 -- 172.21.15.26:0/2235065056 >> 172.21.15.26:0/2235065056 conn(0x7f65d00b7ea0 msgr2=0x7f65d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 1 -- 172.21.15.26:0/2235065056 shutdown_connections 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.437+0000 7f65d4d13640 1 -- 172.21.15.26:0/2235065056 wait complete. 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 10 monclient: build_initial_monmap 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 10 monclient: monmap: 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:23.445 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 1 librados: starting msgr at 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 1 librados: starting objecter 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 1 Processor -- start 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 1 -- start start 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 1 librados: setting wanted keys 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 1 librados: calling monclient init 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.438+0000 7f65d4d13640 10 monclient: init 2024-03-25T14:34:23.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient(hunting): picked mon.noname-a con 0x7f65d00a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient(hunting): picked mon.noname-b con 0x7f65d013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient(hunting): picked mon.noname-c con 0x7f65d01430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d0139330 con 0x7f65d00a98f0 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00abe70 con 0x7f65d013f710 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00ac150 con 0x7f65d01430c0 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:23.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.439+0000 7f65d4d13640 10 monclient(hunting): authenticate will time out at 1948.542252s 2024-03-25T14:34:23.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.443+0000 7f65c7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f65d013f710 0x7f65d00ac570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:54824/0 (socket says 172.21.15.26:54824) 2024-03-25T14:34:23.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.443+0000 7f65c7fff640 1 -- 172.21.15.26:0/3218914824 learned_addr learned my addr 172.21.15.26:0/3218914824 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:23.449 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.444+0000 7f65c6ffd640 10 client.?.objecter ms_handle_connect 0x7f65d013f710 2024-03-25T14:34:23.449 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.444+0000 7f65c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:23.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.450+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998340578 0 0) 0x7f65d00abe70 con 0x7f65d013f710 2024-03-25T14:34:23.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.450+0000 7f65c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:23.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.450+0000 7f65c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:23.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.450+0000 7f65c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:23.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.450+0000 7f65c6ffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T14:34:23.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.450+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b0004eb0 con 0x7f65d013f710 2024-03-25T14:34:23.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.453+0000 7f65c6ffd640 10 client.?.objecter ms_handle_connect 0x7f65d00a98f0 2024-03-25T14:34:23.457 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.453+0000 7f65c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:23.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.455+0000 7f65c6ffd640 10 client.?.objecter ms_handle_connect 0x7f65d01430c0 2024-03-25T14:34:23.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.455+0000 7f65c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:23.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.459+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3800364386 0 0) 0x7f65b0004eb0 con 0x7f65d013f710 2024-03-25T14:34:23.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.459+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65d00abe70 con 0x7f65d013f710 2024-03-25T14:34:23.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.461+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295717847 0 0) 0x7f65d0139330 con 0x7f65d00a98f0 2024-03-25T14:34:23.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.461+0000 7f65c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:23.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.461+0000 7f65c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:23.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.461+0000 7f65c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:23.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.461+0000 7f65c6ffd640 10 monclient(hunting): my global_id is 4560 2024-03-25T14:34:23.467 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.461+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b0004eb0 con 0x7f65d00a98f0 2024-03-25T14:34:23.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.464+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f65b8001970 con 0x7f65d013f710 2024-03-25T14:34:23.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.464+0000 7f65c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:23.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.464+0000 7f65c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.464+0000 7f65c6ffd640 10 monclient(hunting): dump: 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:23.470 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.464+0000 7f65c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.465+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867623951 0 0) 0x7f65d00ac150 con 0x7f65d01430c0 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.465+0000 7f65c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.465+0000 7f65c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:23.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.465+0000 7f65c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:23.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.465+0000 7f65c6ffd640 10 monclient(hunting): my global_id is 4511 2024-03-25T14:34:23.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.465+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d0139330 con 0x7f65d01430c0 2024-03-25T14:34:23.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2840609092 0 0) 0x7f65d00abe70 con 0x7f65d013f710 2024-03-25T14:34:23.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 >> v1:172.21.15.26:6790/0 conn(0x7f65d01430c0 legacy=0x7f65d01e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:23.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 >> v1:172.21.15.26:6789/0 conn(0x7f65d00a98f0 legacy=0x7f65d01389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:23.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:34:23.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 1 monclient: found mon.b 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00acc80 con 0x7f65d013f710 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 5 monclient: authenticate success, global_id 4546 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 10 monclient: _renew_subs 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d0137e30 con 0x7f65d013f710 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 10 monclient: _renew_subs 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d0136880 con 0x7f65d013f710 2024-03-25T14:34:23.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.474+0000 7f65d4d13640 1 librados: init done 2024-03-25T14:34:23.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.475+0000 7f65d4d13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:23.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.475+0000 7f65d4d13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:23.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.475+0000 7f65d4d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:23.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.475+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d00627b0 con 0x7f65d013f710 2024-03-25T14:34:23.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65b80047e0 con 0x7f65d013f710 2024-03-25T14:34:23.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:23.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f65b8004bf0 con 0x7f65d013f710 2024-03-25T14:34:23.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:23.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65c6ffd640 10 monclient: dump: 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:23.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.754+0000 7f65cd59a640 4 set_mon_vals no callback set 2024-03-25T14:34:23.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.755+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f65b8004e10 con 0x7f65d013f710 2024-03-25T14:34:23.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.755+0000 7f65ce59c640 10 monclient: get_auth_request con 0x7f65b006dde0 auth_method 0 2024-03-25T14:34:23.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.756+0000 7f65c6ffd640 10 client.4546.objecter ms_handle_connect 0x7f65b006dde0 2024-03-25T14:34:23.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.759+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f65b80548b0 con 0x7f65d013f710 2024-03-25T14:34:23.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.759+0000 7f65c6ffd640 10 client.4546.objecter ms_dispatch 0x7f65d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:23.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.759+0000 7f65c6ffd640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:23.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.759+0000 7f65c6ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:23.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.759+0000 7f65c6ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T14:34:23.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.759+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65b80503c0 con 0x7f65d013f710 2024-03-25T14:34:23.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.760+0000 7f65c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:23.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:23.760+0000 7f65c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:24.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.015+0000 7f65d4d13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:24.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.016+0000 7f65d4d13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:24.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.016+0000 7f65d4d13640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:24.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.016+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65d01390e0 con 0x7f65d013f710 2024-03-25T14:34:24.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.038+0000 7f65c6ffd640 1 -- 172.21.15.26:0/3218914824 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f65b80522f0 con 0x7f65d013f710 2024-03-25T14:34:24.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.038+0000 7f65c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:24.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.038+0000 7f65c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:24.043 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:24.043 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.040+0000 7f65d4d13640 10 librados: watch_flush enter 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.040+0000 7f65d4d13640 10 librados: watch_flush exit 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.040+0000 7f65d4d13640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.040+0000 7f65d4d13640 10 client.4546.objecter successfully canceled tick 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 >> v1:172.21.15.121:6800/288813066 conn(0x7f65b006dde0 legacy=0x7f65b00702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 10 monclient: shutdown 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 >> v1:172.21.15.121:6789/0 conn(0x7f65d013f710 legacy=0x7f65d00ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:24.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 shutdown_connections 2024-03-25T14:34:24.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 >> 172.21.15.26:0/3218914824 conn(0x7f65d00b7ea0 msgr2=0x7f65d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:24.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 shutdown_connections 2024-03-25T14:34:24.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 -- 172.21.15.26:0/3218914824 wait complete. 2024-03-25T14:34:24.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:24.041+0000 7f65d4d13640 1 librados: shutdown 2024-03-25T14:34:24.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:30.865 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:30.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.973+0000 7f793ecf1640 10 monclient: get_monmap_and_config 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.973+0000 7f793ecf1640 10 monclient: build_initial_monmap 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.973+0000 7f793ecf1640 10 monclient: monmap: 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:30.973982+0000 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:30.973982+0000 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:30.978 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:30.979 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:30.979 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:30.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 1 Processor -- start 2024-03-25T14:34:30.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 1 -- start start 2024-03-25T14:34:30.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient: init 2024-03-25T14:34:30.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:30.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient(hunting): picked mon.noname-c con 0x7f79380a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient(hunting): picked mon.noname-a con 0x7f793813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient(hunting): picked mon.noname-b con 0x7f79381430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7938142520 con 0x7f793813f710 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380b28a0 con 0x7f79380a98f0 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380b2a30 con 0x7f79381430c0 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:30.981 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.974+0000 7f793ecf1640 10 monclient(hunting): authenticate will time out at 1956.077572s 2024-03-25T14:34:30.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.980+0000 7f793ca8d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f79380a98f0 0x7f79380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:34518/0 (socket says 172.21.15.26:34518) 2024-03-25T14:34:30.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.980+0000 7f793ca8d640 1 -- 172.21.15.26:0/4203190456 learned_addr learned my addr 172.21.15.26:0/4203190456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:31.000 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.996+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801312538 0 0) 0x7f79380b28a0 con 0x7f79380a98f0 2024-03-25T14:34:31.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.996+0000 7f79377fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:31.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.996+0000 7f79377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:31.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.996+0000 7f79377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:31.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.996+0000 7f79377fe640 10 monclient(hunting): my global_id is 4549 2024-03-25T14:34:31.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:30.996+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7928003520 con 0x7f79380a98f0 2024-03-25T14:34:31.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.000+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233833541 0 0) 0x7f79380b2a30 con 0x7f79381430c0 2024-03-25T14:34:31.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.000+0000 7f79377fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:31.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.000+0000 7f79377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:31.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.000+0000 7f79377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:31.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.000+0000 7f79377fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T14:34:31.005 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.000+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79380b28a0 con 0x7f79381430c0 2024-03-25T14:34:31.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.013+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280532535 0 0) 0x7f7938142520 con 0x7f793813f710 2024-03-25T14:34:31.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.013+0000 7f79377fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:31.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.013+0000 7f79377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:31.018 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.013+0000 7f79377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:31.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.013+0000 7f79377fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T14:34:31.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.013+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79380b2a30 con 0x7f793813f710 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2550609440 0 0) 0x7f79380b28a0 con 0x7f79381430c0 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 >> v1:172.21.15.121:6789/0 conn(0x7f79380a98f0 legacy=0x7f79380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 >> v1:172.21.15.26:6789/0 conn(0x7f793813f710 legacy=0x7f79381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: _finish_hunting 0 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 1 monclient: found mon.noname-b 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:34:31.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380b44c0 con 0x7f79381430c0 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: _finish_auth 0 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: _check_auth_tickets 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f79300019f0 con 0x7f79381430c0 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f793ecf1640 5 monclient: authenticate success, global_id 4514 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f793ecf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f79377fe640 10 monclient: dump: 2024-03-25T14:34:31.021 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.014+0000 7f793ecf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7930001bd0 con 0x7f79381430c0 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f79377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f79377fe640 1 -- 172.21.15.26:0/4203190456 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f7930001fe0 con 0x7f79381430c0 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f79377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f79377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f79377fe640 10 monclient: dump: 2024-03-25T14:34:31.031 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f793ecf1640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:31.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.026+0000 7f793ecf1640 4 set_mon_vals no callback set 2024-03-25T14:34:31.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 10 monclient: shutdown 2024-03-25T14:34:31.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 1 -- 172.21.15.26:0/4203190456 >> v1:172.21.15.26:6790/0 conn(0x7f79381430c0 legacy=0x7f793814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 1 -- 172.21.15.26:0/4203190456 shutdown_connections 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 1 -- 172.21.15.26:0/4203190456 >> 172.21.15.26:0/4203190456 conn(0x7f79380b7ea0 msgr2=0x7f79380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 1 -- 172.21.15.26:0/4203190456 shutdown_connections 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.027+0000 7f793ecf1640 1 -- 172.21.15.26:0/4203190456 wait complete. 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.028+0000 7f793ecf1640 10 monclient: build_initial_monmap 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.028+0000 7f793ecf1640 10 monclient: monmap: 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:31.036 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.028+0000 7f793ecf1640 1 librados: starting msgr at 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.028+0000 7f793ecf1640 1 librados: starting objecter 2024-03-25T14:34:31.037 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 1 Processor -- start 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 1 -- start start 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 1 librados: setting wanted keys 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 1 librados: calling monclient init 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 10 monclient: init 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 10 monclient(hunting): picked mon.noname-a con 0x7f79380a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 10 monclient(hunting): picked mon.noname-c con 0x7f793813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 10 monclient(hunting): picked mon.noname-b con 0x7f79381430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.029+0000 7f793ecf1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380abac0 con 0x7f79380a98f0 2024-03-25T14:34:31.038 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.030+0000 7f793ecf1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79381d4310 con 0x7f79381430c0 2024-03-25T14:34:31.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.030+0000 7f793ecf1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7938138f10 con 0x7f793813f710 2024-03-25T14:34:31.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.030+0000 7f793ecf1640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:31.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.030+0000 7f793ecf1640 10 monclient(hunting): authenticate will time out at 1956.132750s 2024-03-25T14:34:31.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.047+0000 7f793ca8d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f79380a98f0 0x7f7938138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47126/0 (socket says 172.21.15.26:47126) 2024-03-25T14:34:31.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.047+0000 7f793ca8d640 1 -- 172.21.15.26:0/3571874077 learned_addr learned my addr 172.21.15.26:0/3571874077 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:31.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.048+0000 7f79357fa640 10 client.?.objecter ms_handle_connect 0x7f79380a98f0 2024-03-25T14:34:31.053 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.048+0000 7f79357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:31.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.055+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713822106 0 0) 0x7f79380abac0 con 0x7f79380a98f0 2024-03-25T14:34:31.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.055+0000 7f79357fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:31.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.055+0000 7f79357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:31.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.055+0000 7f79357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:31.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.055+0000 7f79357fa640 10 monclient(hunting): my global_id is 4566 2024-03-25T14:34:31.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.055+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f791c004eb0 con 0x7f79380a98f0 2024-03-25T14:34:31.062 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.057+0000 7f79357fa640 10 client.?.objecter ms_handle_connect 0x7f793813f710 2024-03-25T14:34:31.062 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.057+0000 7f79357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:31.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571744053 0 0) 0x7f7938138f10 con 0x7f793813f710 2024-03-25T14:34:31.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:31.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:31.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:31.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 10 monclient(hunting): my global_id is 4517 2024-03-25T14:34:31.068 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79380abac0 con 0x7f793813f710 2024-03-25T14:34:31.068 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.062+0000 7f79357fa640 10 client.?.objecter ms_handle_connect 0x7f79381430c0 2024-03-25T14:34:31.068 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.063+0000 7f79357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 665459990 0 0) 0x7f791c004eb0 con 0x7f79380a98f0 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7938138f10 con 0x7f79380a98f0 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f792c001950 con 0x7f79380a98f0 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 10 monclient(hunting): dump: 2024-03-25T14:34:31.072 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.067+0000 7f79357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:31.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.084+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269664175 0 0) 0x7f79381d4310 con 0x7f79381430c0 2024-03-25T14:34:31.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.084+0000 7f79357fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:31.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.084+0000 7f79357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:31.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.084+0000 7f79357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:31.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.084+0000 7f79357fa640 10 monclient(hunting): my global_id is 4552 2024-03-25T14:34:31.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.084+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f791c004eb0 con 0x7f79381430c0 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.099+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 398851784 0 0) 0x7f79380abac0 con 0x7f793813f710 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.099+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79381d4310 con 0x7f793813f710 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.099+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f79240018b0 con 0x7f793813f710 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.099+0000 7f79357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.099+0000 7f79357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.099+0000 7f79357fa640 10 monclient(hunting): dump: 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:31.104 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:31.105 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.102+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1274805750 0 0) 0x7f791c004eb0 con 0x7f79381430c0 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.102+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79380abac0 con 0x7f79381430c0 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.102+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f7930001ee0 con 0x7f79381430c0 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.102+0000 7f79357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.102+0000 7f79357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.102+0000 7f79357fa640 10 monclient(hunting): dump: 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:31.107 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:31.108 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:31.108 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:31.108 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:31.108 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900151415 0 0) 0x7f7938138f10 con 0x7f79380a98f0 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 >> v1:172.21.15.26:6790/0 conn(0x7f793813f710 legacy=0x7f7938134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 >> v1:172.21.15.121:6789/0 conn(0x7f79381430c0 legacy=0x7f79381d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 10 monclient: _finish_hunting 0 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 1 monclient: found mon.a 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7938135010 con 0x7f79380a98f0 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 10 monclient: _finish_auth 0 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 10 monclient: _check_auth_tickets 2024-03-25T14:34:31.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f79357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 5 monclient: authenticate success, global_id 4566 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 10 monclient: _renew_subs 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7938134790 con 0x7f79380a98f0 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 10 monclient: _renew_subs 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7938134a40 con 0x7f79380a98f0 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.103+0000 7f793ecf1640 1 librados: init done 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.104+0000 7f793ecf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.104+0000 7f793ecf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.104+0000 7f793ecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:34:31.111 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.104+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79381d44a0 con 0x7f79380a98f0 2024-03-25T14:34:31.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f792c0047e0 con 0x7f79380a98f0 2024-03-25T14:34:31.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f79357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:31.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f792c004af0 con 0x7f79380a98f0 2024-03-25T14:34:31.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f79357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:31.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f79357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f79357fa640 10 monclient: dump: 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:31.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.447+0000 7f7936ffd640 4 set_mon_vals no callback set 2024-03-25T14:34:31.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.448+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f792c01aac0 con 0x7f79380a98f0 2024-03-25T14:34:31.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.449+0000 7f7937fff640 10 monclient: get_auth_request con 0x7f791c06e110 auth_method 0 2024-03-25T14:34:31.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.449+0000 7f79357fa640 10 client.4566.objecter ms_handle_connect 0x7f791c06e110 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f792c0547f0 con 0x7f79380a98f0 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 10 client.4566.objecter ms_dispatch 0x7f7938000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f792c050130 con 0x7f79380a98f0 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:31.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.454+0000 7f79357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:31.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.709+0000 7f793ecf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:31.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.709+0000 7f793ecf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:31.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.709+0000 7f793ecf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:34:31.714 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.709+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79381351a0 con 0x7f79380a98f0 2024-03-25T14:34:31.739 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.734+0000 7f79357fa640 1 -- 172.21.15.26:0/3571874077 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f792c052230 con 0x7f79380a98f0 2024-03-25T14:34:31.739 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.734+0000 7f79357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:31.739 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.734+0000 7f79357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:31.739 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:31.739 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 10 librados: watch_flush enter 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 10 librados: watch_flush exit 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 10 client.4566.objecter successfully canceled tick 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 >> v1:172.21.15.121:6800/288813066 conn(0x7f791c06e110 legacy=0x7f791c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 10 monclient: shutdown 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 >> v1:172.21.15.26:6789/0 conn(0x7f79380a98f0 legacy=0x7f7938138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:31.742 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 shutdown_connections 2024-03-25T14:34:31.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 >> 172.21.15.26:0/3571874077 conn(0x7f79380b7ea0 msgr2=0x7f7938074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:31.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 shutdown_connections 2024-03-25T14:34:31.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 -- 172.21.15.26:0/3571874077 wait complete. 2024-03-25T14:34:31.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:31.737+0000 7f793ecf1640 1 librados: shutdown 2024-03-25T14:34:31.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:38.561 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:38.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.674+0000 7fc708fe9640 10 monclient: get_monmap_and_config 2024-03-25T14:34:38.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.674+0000 7fc708fe9640 10 monclient: build_initial_monmap 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.674+0000 7fc708fe9640 10 monclient: monmap: 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:38.674908+0000 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:38.674908+0000 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:38.679 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:38.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.674+0000 7fc708fe9640 1 Processor -- start 2024-03-25T14:34:38.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.674+0000 7fc708fe9640 1 -- start start 2024-03-25T14:34:38.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.674+0000 7fc708fe9640 10 monclient: init 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient(hunting): picked mon.noname-a con 0x7fc7040a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient(hunting): picked mon.noname-c con 0x7fc70413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient(hunting): picked mon.noname-b con 0x7fc7041430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704142520 con 0x7fc7040a98f0 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b28a0 con 0x7fc70413f710 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b2a30 con 0x7fc7041430c0 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:38.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.675+0000 7fc708fe9640 10 monclient(hunting): authenticate will time out at 1963.778156s 2024-03-25T14:34:38.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.677+0000 7fc70259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc7040a98f0 0x7fc7040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39086/0 (socket says 172.21.15.26:39086) 2024-03-25T14:34:38.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.677+0000 7fc70259c640 1 -- 172.21.15.26:0/2447549490 learned_addr learned my addr 172.21.15.26:0/2447549490 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:38.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.690+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76431310 0 0) 0x7fc704142520 con 0x7fc7040a98f0 2024-03-25T14:34:38.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.690+0000 7fc70159a640 10 monclient(hunting): _negotiate 2024-03-25T14:34:38.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.690+0000 7fc70159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:38.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.690+0000 7fc70159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:38.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.690+0000 7fc70159a640 10 monclient(hunting): my global_id is 4569 2024-03-25T14:34:38.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.690+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003520 con 0x7fc7040a98f0 2024-03-25T14:34:38.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.693+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908180483 0 0) 0x7fc7040b28a0 con 0x7fc70413f710 2024-03-25T14:34:38.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.693+0000 7fc70159a640 10 monclient(hunting): _negotiate 2024-03-25T14:34:38.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.693+0000 7fc70159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:38.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.693+0000 7fc70159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:38.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.693+0000 7fc70159a640 10 monclient(hunting): my global_id is 4555 2024-03-25T14:34:38.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.693+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc704142520 con 0x7fc70413f710 2024-03-25T14:34:38.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.694+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599054928 0 0) 0x7fc7040b2a30 con 0x7fc7041430c0 2024-03-25T14:34:38.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.694+0000 7fc70159a640 10 monclient(hunting): _negotiate 2024-03-25T14:34:38.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.694+0000 7fc70159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:38.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.694+0000 7fc70159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:38.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.694+0000 7fc70159a640 10 monclient(hunting): my global_id is 4520 2024-03-25T14:34:38.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.694+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7040b28a0 con 0x7fc7041430c0 2024-03-25T14:34:38.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3207553999 0 0) 0x7fc704142520 con 0x7fc70413f710 2024-03-25T14:34:38.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 >> v1:172.21.15.26:6790/0 conn(0x7fc7041430c0 legacy=0x7fc70414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:38.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 >> v1:172.21.15.26:6789/0 conn(0x7fc7040a98f0 legacy=0x7fc7040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:38.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: _finish_hunting 0 2024-03-25T14:34:38.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 1 monclient: found mon.noname-c 2024-03-25T14:34:38.704 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040b44c0 con 0x7fc70413f710 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: _finish_auth 0 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: _check_auth_tickets 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc6f8001670 con 0x7fc70413f710 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc70159a640 10 monclient: dump: 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:38.705 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc708fe9640 5 monclient: authenticate success, global_id 4555 2024-03-25T14:34:38.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.699+0000 7fc708fe9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:38.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.702+0000 7fc70159a640 1 -- 172.21.15.26:0/2447549490 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6f80017d0 con 0x7fc70413f710 2024-03-25T14:34:38.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.702+0000 7fc70159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:38.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.702+0000 7fc708fe9640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:38.707 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.702+0000 7fc708fe9640 4 set_mon_vals no callback set 2024-03-25T14:34:38.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 10 monclient: shutdown 2024-03-25T14:34:38.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:38.708 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 1 -- 172.21.15.26:0/2447549490 >> v1:172.21.15.121:6789/0 conn(0x7fc70413f710 legacy=0x7fc7041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 1 -- 172.21.15.26:0/2447549490 shutdown_connections 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 1 -- 172.21.15.26:0/2447549490 >> 172.21.15.26:0/2447549490 conn(0x7fc7040b7ea0 msgr2=0x7fc7040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 1 -- 172.21.15.26:0/2447549490 shutdown_connections 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.703+0000 7fc708fe9640 1 -- 172.21.15.26:0/2447549490 wait complete. 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.705+0000 7fc708fe9640 10 monclient: build_initial_monmap 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.705+0000 7fc708fe9640 10 monclient: monmap: 2024-03-25T14:34:38.709 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:38.710 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.705+0000 7fc708fe9640 1 librados: starting msgr at 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.705+0000 7fc708fe9640 1 librados: starting objecter 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.705+0000 7fc708fe9640 1 Processor -- start 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 1 -- start start 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 1 librados: setting wanted keys 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 1 librados: calling monclient init 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 10 monclient: init 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 10 monclient(hunting): picked mon.noname-b con 0x7fc7040a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:38.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 10 monclient(hunting): picked mon.noname-c con 0x7fc70413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:38.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 10 monclient(hunting): picked mon.noname-a con 0x7fc7041430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:38.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704137f20 con 0x7fc7041430c0 2024-03-25T14:34:38.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.706+0000 7fc708fe9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7041387a0 con 0x7fc7040a98f0 2024-03-25T14:34:38.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.707+0000 7fc708fe9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b17b0 con 0x7fc70413f710 2024-03-25T14:34:38.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.707+0000 7fc708fe9640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:38.713 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.707+0000 7fc708fe9640 10 monclient(hunting): authenticate will time out at 1963.809613s 2024-03-25T14:34:38.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.715+0000 7fc702d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc7041430c0 0x7fc7041daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39136/0 (socket says 172.21.15.26:39136) 2024-03-25T14:34:38.720 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.715+0000 7fc702d9d640 1 -- 172.21.15.26:0/4203308338 learned_addr learned my addr 172.21.15.26:0/4203308338 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:38.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.719+0000 7fc6f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7041430c0 2024-03-25T14:34:38.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.719+0000 7fc6f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:38.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.719+0000 7fc6f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7040a98f0 2024-03-25T14:34:38.724 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.719+0000 7fc6f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:38.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.726+0000 7fc6f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc70413f710 2024-03-25T14:34:38.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.726+0000 7fc6f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:38.735 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.731+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365651932 0 0) 0x7fc704137f20 con 0x7fc7041430c0 2024-03-25T14:34:38.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.731+0000 7fc6f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:38.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.731+0000 7fc6f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:38.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.731+0000 7fc6f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:38.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.731+0000 7fc6f2ffd640 10 monclient(hunting): my global_id is 4572 2024-03-25T14:34:38.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.731+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e4004eb0 con 0x7fc7041430c0 2024-03-25T14:34:38.736 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.732+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71682703 0 0) 0x7fc7041387a0 con 0x7fc7040a98f0 2024-03-25T14:34:38.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.732+0000 7fc6f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:38.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.732+0000 7fc6f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:38.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.732+0000 7fc6f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:38.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.732+0000 7fc6f2ffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T14:34:38.737 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.732+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc704137f20 con 0x7fc7040a98f0 2024-03-25T14:34:38.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943155463 0 0) 0x7fc704137f20 con 0x7fc7040a98f0 2024-03-25T14:34:38.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7041387a0 con 0x7fc7040a98f0 2024-03-25T14:34:38.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc6f4001950 con 0x7fc7040a98f0 2024-03-25T14:34:38.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:38.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:38.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): dump: 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61063539 0 0) 0x7fc7040b17b0 con 0x7fc70413f710 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:34:38.750 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:38.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:38.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 10 monclient(hunting): my global_id is 4523 2024-03-25T14:34:38.751 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.744+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc704137f20 con 0x7fc70413f710 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4259652031 0 0) 0x7fc7041387a0 con 0x7fc7040a98f0 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 >> v1:172.21.15.26:6790/0 conn(0x7fc70413f710 legacy=0x7fc7041da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 >> v1:172.21.15.26:6789/0 conn(0x7fc7041430c0 legacy=0x7fc7041daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 1 monclient: found mon.b 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7041380d0 con 0x7fc7040a98f0 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 10 monclient: _finish_auth 0 2024-03-25T14:34:38.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc6f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 5 monclient: authenticate success, global_id 4558 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 10 monclient: _renew_subs 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc704138320 con 0x7fc7040a98f0 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 10 monclient: _renew_subs 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7041e3020 con 0x7fc7040a98f0 2024-03-25T14:34:38.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.754+0000 7fc708fe9640 1 librados: init done 2024-03-25T14:34:38.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.755+0000 7fc708fe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:38.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.755+0000 7fc708fe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:38.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.755+0000 7fc708fe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:38.761 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:38.755+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7040b3ef0 con 0x7fc7040a98f0 2024-03-25T14:34:39.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6f4001d60 con 0x7fc7040a98f0 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc6f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc6f4004980 con 0x7fc7040a98f0 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc6f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc6f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc6f2ffd640 10 monclient: dump: 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:39.030 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:39.031 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:39.031 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:39.031 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:39.031 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:39.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.025+0000 7fc700d99640 4 set_mon_vals no callback set 2024-03-25T14:34:39.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.026+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fc6f4004ba0 con 0x7fc7040a98f0 2024-03-25T14:34:39.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.026+0000 7fc701d9b640 10 monclient: get_auth_request con 0x7fc6e406ddc0 auth_method 0 2024-03-25T14:34:39.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.026+0000 7fc6f2ffd640 10 client.4558.objecter ms_handle_connect 0x7fc6e406ddc0 2024-03-25T14:34:39.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.031+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fc6f40547d0 con 0x7fc7040a98f0 2024-03-25T14:34:39.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.031+0000 7fc6f2ffd640 10 client.4558.objecter ms_dispatch 0x7fc704000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:39.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.031+0000 7fc6f2ffd640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:39.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.031+0000 7fc6f2ffd640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:39.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.031+0000 7fc6f2ffd640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T14:34:39.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.032+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6f4050110 con 0x7fc7040a98f0 2024-03-25T14:34:39.037 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.032+0000 7fc6f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:39.037 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.032+0000 7fc6f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:39.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.287+0000 7fc708fe9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:39.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.287+0000 7fc708fe9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:39.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.287+0000 7fc708fe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:34:39.292 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.287+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7040627b0 con 0x7fc7040a98f0 2024-03-25T14:34:39.419 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.414+0000 7fc6f2ffd640 1 -- 172.21.15.26:0/4203308338 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fc6f4052210 con 0x7fc7040a98f0 2024-03-25T14:34:39.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.415+0000 7fc6f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:39.420 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.415+0000 7fc6f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:39.420 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:39.420 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 10 librados: watch_flush enter 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 10 librados: watch_flush exit 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 10 client.4558.objecter successfully canceled tick 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 >> v1:172.21.15.121:6800/288813066 conn(0x7fc6e406ddc0 legacy=0x7fc6e4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 10 monclient: shutdown 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 >> v1:172.21.15.121:6789/0 conn(0x7fc7040a98f0 legacy=0x7fc704137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:39.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 shutdown_connections 2024-03-25T14:34:39.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 >> 172.21.15.26:0/4203308338 conn(0x7fc7040b7ea0 msgr2=0x7fc704074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:39.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 shutdown_connections 2024-03-25T14:34:39.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 -- 172.21.15.26:0/4203308338 wait complete. 2024-03-25T14:34:39.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:39.417+0000 7fc708fe9640 1 librados: shutdown 2024-03-25T14:34:39.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:42.104 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:00.965 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.157 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:01.018 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.196 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:01.057 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.219 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:01.096 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.228 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:01.088 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.234 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:01.111 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.237 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:01.098 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.272 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:01.132 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:42.272 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:01.149 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:34:46.241 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:46.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.351+0000 7f548ef3d640 10 monclient: get_monmap_and_config 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.351+0000 7f548ef3d640 10 monclient: build_initial_monmap 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.351+0000 7f548ef3d640 10 monclient: monmap: 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:46.352114+0000 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:46.352114+0000 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:46.356 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 1 Processor -- start 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 1 -- start start 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 10 monclient: init 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 10 monclient(hunting): picked mon.noname-a con 0x7f54880b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 10 monclient(hunting): picked mon.noname-c con 0x7f54880b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 10 monclient(hunting): picked mon.noname-b con 0x7f548814be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:46.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54880af940 con 0x7f54880b4d60 2024-03-25T14:34:46.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.352+0000 7f548ef3d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54880b0a70 con 0x7f54880b1730 2024-03-25T14:34:46.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.353+0000 7f548ef3d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54880b0d50 con 0x7f548814be90 2024-03-25T14:34:46.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.353+0000 7f548ef3d640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:46.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.353+0000 7f548ef3d640 10 monclient(hunting): authenticate will time out at 1971.455646s 2024-03-25T14:34:46.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.354+0000 7f5487fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f54880b1730 0x7f54880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:52220/0 (socket says 172.21.15.26:52220) 2024-03-25T14:34:46.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.354+0000 7f5487fff640 1 -- 172.21.15.26:0/2084453342 learned_addr learned my addr 172.21.15.26:0/2084453342 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:46.415 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.411+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612380930 0 0) 0x7f54880af940 con 0x7f54880b4d60 2024-03-25T14:34:46.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.411+0000 7f54877fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:46.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.411+0000 7f54877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:46.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.411+0000 7f54877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:46.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.411+0000 7f54877fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T14:34:46.416 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.411+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5478003520 con 0x7f54880b4d60 2024-03-25T14:34:46.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.417+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482979584 0 0) 0x7f54880b0d50 con 0x7f548814be90 2024-03-25T14:34:46.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.417+0000 7f54877fe640 10 monclient(hunting): _negotiate 2024-03-25T14:34:46.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.417+0000 7f54877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:46.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.417+0000 7f54877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:46.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.417+0000 7f54877fe640 10 monclient(hunting): my global_id is 4526 2024-03-25T14:34:46.422 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.417+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54880af940 con 0x7f548814be90 2024-03-25T14:34:46.423 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.418+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727845399 0 0) 0x7f5478003520 con 0x7f54880b4d60 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.418+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 >> v1:172.21.15.26:6790/0 conn(0x7f548814be90 legacy=0x7f548814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.418+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 >> v1:172.21.15.121:6789/0 conn(0x7f54880b1730 legacy=0x7f54880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.418+0000 7f54877fe640 10 monclient: _finish_hunting 0 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.418+0000 7f54877fe640 1 monclient: found mon.noname-a 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.418+0000 7f54877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54880b1030 con 0x7f54880b4d60 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 10 monclient: _finish_auth 0 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 10 monclient: _check_auth_tickets 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f547c001690 con 0x7f54880b4d60 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f548ef3d640 5 monclient: authenticate success, global_id 4575 2024-03-25T14:34:46.424 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f548ef3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f54877fe640 10 monclient: dump: 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:46.425 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.419+0000 7f548ef3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f547c0017f0 con 0x7f54880b4d60 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f54877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f54877fe640 1 -- 172.21.15.26:0/2084453342 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f547c001c00 con 0x7f54880b4d60 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f548ef3d640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f548ef3d640 4 set_mon_vals no callback set 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f54877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f54877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.435+0000 7f54877fe640 10 monclient: dump: 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:46.440 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:46.441 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:46.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 10 monclient: shutdown 2024-03-25T14:34:46.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:46.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 1 -- 172.21.15.26:0/2084453342 >> v1:172.21.15.26:6789/0 conn(0x7f54880b4d60 legacy=0x7f548813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:46.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 1 -- 172.21.15.26:0/2084453342 shutdown_connections 2024-03-25T14:34:46.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 1 -- 172.21.15.26:0/2084453342 >> 172.21.15.26:0/2084453342 conn(0x7f54880617a0 msgr2=0x7f5488061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 1 -- 172.21.15.26:0/2084453342 shutdown_connections 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.436+0000 7f548ef3d640 1 -- 172.21.15.26:0/2084453342 wait complete. 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 10 monclient: build_initial_monmap 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 10 monclient: monmap: 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:46.443 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 1 librados: starting msgr at 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 1 librados: starting objecter 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 1 Processor -- start 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 1 -- start start 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 1 librados: setting wanted keys 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 1 librados: calling monclient init 2024-03-25T14:34:46.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.437+0000 7f548ef3d640 10 monclient: init 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient(hunting): picked mon.noname-b con 0x7f54880b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient(hunting): picked mon.noname-c con 0x7f54880b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient(hunting): picked mon.noname-a con 0x7f548814be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5488143690 con 0x7f548814be90 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548813ee90 con 0x7f54880b1730 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548813f020 con 0x7f54880b4d60 2024-03-25T14:34:46.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:46.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.438+0000 7f548ef3d640 10 monclient(hunting): authenticate will time out at 1971.541212s 2024-03-25T14:34:46.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.448+0000 7f5487fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f54880b4d60 0x7f54881d7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59070/0 (socket says 172.21.15.26:59070) 2024-03-25T14:34:46.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.448+0000 7f5487fff640 1 -- 172.21.15.26:0/3277855837 learned_addr learned my addr 172.21.15.26:0/3277855837 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:46.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.451+0000 7f54857fa640 10 client.?.objecter ms_handle_connect 0x7f54880b4d60 2024-03-25T14:34:46.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.451+0000 7f54857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:46.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.458+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163149295 0 0) 0x7f548813f020 con 0x7f54880b4d60 2024-03-25T14:34:46.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.458+0000 7f54857fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:46.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.458+0000 7f54857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:46.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.458+0000 7f54857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:46.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.458+0000 7f54857fa640 10 monclient(hunting): my global_id is 4529 2024-03-25T14:34:46.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.458+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f546c004eb0 con 0x7f54880b4d60 2024-03-25T14:34:46.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.464+0000 7f54857fa640 10 client.?.objecter ms_handle_connect 0x7f548814be90 2024-03-25T14:34:46.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.464+0000 7f54857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:46.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.472+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349714579 0 0) 0x7f5488143690 con 0x7f548814be90 2024-03-25T14:34:46.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.472+0000 7f54857fa640 10 monclient(hunting): _negotiate 2024-03-25T14:34:46.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.472+0000 7f54857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:46.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.472+0000 7f54857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:46.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.472+0000 7f54857fa640 10 monclient(hunting): my global_id is 4578 2024-03-25T14:34:46.477 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.472+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548813f020 con 0x7f548814be90 2024-03-25T14:34:46.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147011526 0 0) 0x7f546c004eb0 con 0x7f54880b4d60 2024-03-25T14:34:46.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5488143690 con 0x7f54880b4d60 2024-03-25T14:34:46.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f54740018b0 con 0x7f54880b4d60 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 10 monclient(hunting): dump: 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:46.479 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:46.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.473+0000 7f54857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:46.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.493+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 459030413 0 0) 0x7f548813f020 con 0x7f548814be90 2024-03-25T14:34:46.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.493+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f546c004eb0 con 0x7f548814be90 2024-03-25T14:34:46.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.494+0000 7f54857fa640 10 client.?.objecter ms_handle_connect 0x7f54880b1730 2024-03-25T14:34:46.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.494+0000 7f54857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 736687989 0 0) 0x7f5488143690 con 0x7f54880b4d60 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 >> v1:172.21.15.121:6789/0 conn(0x7f54880b1730 legacy=0x7f548813b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 >> v1:172.21.15.26:6789/0 conn(0x7f548814be90 legacy=0x7f54881d8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 10 monclient: _finish_hunting 0 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 1 monclient: found mon.c 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:46.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.498+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548813bb30 con 0x7f54880b4d60 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f54857fa640 10 monclient: _finish_auth 0 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f54857fa640 10 monclient: _check_auth_tickets 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f54857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 5 monclient: authenticate success, global_id 4529 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 monclient: _renew_subs 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548813bd80 con 0x7f54880b4d60 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 monclient: _renew_subs 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5488143f10 con 0x7f54880b4d60 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 1 librados: init done 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:46.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:46.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:46.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.499+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54880b3ca0 con 0x7f54880b4d60 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54740047e0 con 0x7f54880b4d60 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f54857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5474004c30 con 0x7f54880b4d60 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f54857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f54857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f54857fa640 10 monclient: dump: 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:46.831 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:46.832 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.826+0000 7f5486ffd640 4 set_mon_vals no callback set 2024-03-25T14:34:46.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.827+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f5474004e50 con 0x7f54880b4d60 2024-03-25T14:34:46.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.827+0000 7f548ccd9640 10 monclient: get_auth_request con 0x7f546c06df80 auth_method 0 2024-03-25T14:34:46.833 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.828+0000 7f54857fa640 10 client.4529.objecter ms_handle_connect 0x7f546c06df80 2024-03-25T14:34:46.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f5474054760 con 0x7f54880b4d60 2024-03-25T14:34:46.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 10 client.4529.objecter ms_dispatch 0x7f5488000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:46.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:46.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:46.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-25T14:34:46.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5474050270 con 0x7f54880b4d60 2024-03-25T14:34:46.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:46.839 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:46.833+0000 7f54857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:47.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.094+0000 7f548ef3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:47.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.094+0000 7f548ef3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:47.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.094+0000 7f548ef3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:47.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.094+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54880627b0 con 0x7f54880b4d60 2024-03-25T14:34:47.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.127+0000 7f54857fa640 1 -- 172.21.15.26:0/3277855837 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f54740521a0 con 0x7f54880b4d60 2024-03-25T14:34:47.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.127+0000 7f54857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:47.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.127+0000 7f54857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:47.132 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:47.133 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:47.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.129+0000 7f548ef3d640 10 librados: watch_flush enter 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 10 librados: watch_flush exit 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 10 client.4529.objecter successfully canceled tick 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 >> v1:172.21.15.121:6800/288813066 conn(0x7f546c06df80 legacy=0x7f546c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 10 monclient: shutdown 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 >> v1:172.21.15.26:6790/0 conn(0x7f54880b4d60 legacy=0x7f54881d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 shutdown_connections 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 >> 172.21.15.26:0/3277855837 conn(0x7f54880617a0 msgr2=0x7f548813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 shutdown_connections 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 -- 172.21.15.26:0/3277855837 wait complete. 2024-03-25T14:34:47.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:47.130+0000 7f548ef3d640 1 librados: shutdown 2024-03-25T14:34:47.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:34:53.955 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:34:54.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.065+0000 7f0497433640 10 monclient: get_monmap_and_config 2024-03-25T14:34:54.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.065+0000 7f0497433640 10 monclient: build_initial_monmap 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.065+0000 7f0497433640 10 monclient: monmap: 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:34:54.066936+0000 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:34:54.066936+0000 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:34:54.071 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.065+0000 7f0497433640 1 Processor -- start 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 1 -- start start 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient: init 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient(hunting): picked mon.noname-c con 0x7f04900a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:54.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient(hunting): picked mon.noname-b con 0x7f049013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:54.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient(hunting): picked mon.noname-a con 0x7f04901430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:54.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0490142520 con 0x7f04901430c0 2024-03-25T14:34:54.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900b28a0 con 0x7f04900a98f0 2024-03-25T14:34:54.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900b2a30 con 0x7f049013f710 2024-03-25T14:34:54.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:54.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.066+0000 7f0497433640 10 monclient(hunting): authenticate will time out at 1979.170334s 2024-03-25T14:34:54.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.069+0000 7f04949ce640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f049013f710 0x7f04901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45174/0 (socket says 172.21.15.26:45174) 2024-03-25T14:34:54.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.069+0000 7f04949ce640 1 -- 172.21.15.26:0/1553984162 learned_addr learned my addr 172.21.15.26:0/1553984162 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:54.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.076+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916168808 0 0) 0x7f04900b2a30 con 0x7f049013f710 2024-03-25T14:34:54.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.076+0000 7f0487fff640 10 monclient(hunting): _negotiate 2024-03-25T14:34:54.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.076+0000 7f0487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:54.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.076+0000 7f0487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:54.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.076+0000 7f0487fff640 10 monclient(hunting): my global_id is 4532 2024-03-25T14:34:54.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.076+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f047c003520 con 0x7f049013f710 2024-03-25T14:34:54.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.086+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890156084 0 0) 0x7f04900b28a0 con 0x7f04900a98f0 2024-03-25T14:34:54.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.086+0000 7f0487fff640 10 monclient(hunting): _negotiate 2024-03-25T14:34:54.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.086+0000 7f0487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:54.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.086+0000 7f0487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:54.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.086+0000 7f0487fff640 10 monclient(hunting): my global_id is 4564 2024-03-25T14:34:54.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.086+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04900b2a30 con 0x7f04900a98f0 2024-03-25T14:34:54.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.096+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183324931 0 0) 0x7f0490142520 con 0x7f04901430c0 2024-03-25T14:34:54.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.096+0000 7f0487fff640 10 monclient(hunting): _negotiate 2024-03-25T14:34:54.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.096+0000 7f0487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:54.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.096+0000 7f0487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:54.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.096+0000 7f0487fff640 10 monclient(hunting): my global_id is 4581 2024-03-25T14:34:54.102 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.096+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04900b28a0 con 0x7f04901430c0 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091883558 0 0) 0x7f047c003520 con 0x7f049013f710 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 >> v1:172.21.15.121:6789/0 conn(0x7f04900a98f0 legacy=0x7f04900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 >> v1:172.21.15.26:6789/0 conn(0x7f04901430c0 legacy=0x7f049014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: _finish_hunting 0 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 1 monclient: found mon.noname-b 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:54.105 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04900b44c0 con 0x7f049013f710 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: _finish_auth 0 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: _check_auth_tickets 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0480001690 con 0x7f049013f710 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0497433640 5 monclient: authenticate success, global_id 4532 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0497433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0487fff640 10 monclient: dump: 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.106 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.099+0000 7f0497433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:34:54.118 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04800017f0 con 0x7f049013f710 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0487fff640 1 -- 172.21.15.26:0/1553984162 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0480001c00 con 0x7f049013f710 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0497433640 10 monclient: get_monmap_and_config success 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0497433640 4 set_mon_vals no callback set 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0487fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.113+0000 7f0487fff640 10 monclient: dump: 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:54.119 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.120 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:54.120 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:54.120 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:54.120 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 10 monclient: shutdown 2024-03-25T14:34:54.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:54.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 1 -- 172.21.15.26:0/1553984162 >> v1:172.21.15.26:6790/0 conn(0x7f049013f710 legacy=0x7f04901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 1 -- 172.21.15.26:0/1553984162 shutdown_connections 2024-03-25T14:34:54.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 1 -- 172.21.15.26:0/1553984162 >> 172.21.15.26:0/1553984162 conn(0x7f04900b7ea0 msgr2=0x7f04900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:54.122 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 1 -- 172.21.15.26:0/1553984162 shutdown_connections 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.114+0000 7f0497433640 1 -- 172.21.15.26:0/1553984162 wait complete. 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 10 monclient: build_initial_monmap 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 10 monclient: monmap: 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.123 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 1 librados: starting msgr at 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 1 librados: starting objecter 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 1 Processor -- start 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 1 -- start start 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 1 librados: setting wanted keys 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 1 librados: calling monclient init 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.115+0000 7f0497433640 10 monclient: init 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient: _reopen_session rank -1 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient(hunting): picked mon.noname-b con 0x7f04900a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient(hunting): picked mon.noname-a con 0x7f049013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient(hunting): picked mon.noname-c con 0x7f04901430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900aba00 con 0x7f049013f710 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04901d7490 con 0x7f04900a98f0 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04901d8360 con 0x7f04901430c0 2024-03-25T14:34:54.124 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient(hunting): _renew_subs 2024-03-25T14:34:54.125 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.116+0000 7f0497433640 10 monclient(hunting): authenticate will time out at 1979.220028s 2024-03-25T14:34:54.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.127+0000 7f04949ce640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f049013f710 0x7f04901d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:44480/0 (socket says 172.21.15.26:44480) 2024-03-25T14:34:54.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.127+0000 7f04949ce640 1 -- 172.21.15.26:0/267901716 learned_addr learned my addr 172.21.15.26:0/267901716 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:34:54.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.127+0000 7f0485ffb640 10 client.?.objecter ms_handle_connect 0x7f049013f710 2024-03-25T14:34:54.133 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.127+0000 7f0485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:54.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.130+0000 7f0485ffb640 10 client.?.objecter ms_handle_connect 0x7f04900a98f0 2024-03-25T14:34:54.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.130+0000 7f0485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:54.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.134+0000 7f0485ffb640 10 client.?.objecter ms_handle_connect 0x7f04901430c0 2024-03-25T14:34:54.139 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.134+0000 7f0485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.145+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904075129 0 0) 0x7f04901d7490 con 0x7f04900a98f0 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.145+0000 7f0485ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.145+0000 7f0485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.145+0000 7f0485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.145+0000 7f0485ffb640 10 monclient(hunting): my global_id is 4567 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.145+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0474004eb0 con 0x7f04900a98f0 2024-03-25T14:34:54.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.146+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22671595 0 0) 0x7f04900aba00 con 0x7f049013f710 2024-03-25T14:34:54.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.146+0000 7f0485ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:34:54.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.146+0000 7f0485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:54.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.146+0000 7f0485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:54.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.146+0000 7f0485ffb640 10 monclient(hunting): my global_id is 4584 2024-03-25T14:34:54.152 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.146+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04901d7490 con 0x7f049013f710 2024-03-25T14:34:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.158+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015368963 0 0) 0x7f04901d8360 con 0x7f04901430c0 2024-03-25T14:34:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.158+0000 7f0485ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:34:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.158+0000 7f0485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:34:54.163 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.158+0000 7f0485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:34:54.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.158+0000 7f0485ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T14:34:54.164 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.158+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04900aba00 con 0x7f04901430c0 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4005481591 0 0) 0x7f0474004eb0 con 0x7f04900a98f0 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04901d8360 con 0x7f04900a98f0 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f048c001970 con 0x7f04900a98f0 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 10 monclient(hunting): dump: 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.169 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:54.170 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:54.170 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:54.170 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.163+0000 7f0485ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:34:54.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.167+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871134133 0 0) 0x7f04901d7490 con 0x7f049013f710 2024-03-25T14:34:54.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.167+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0474004eb0 con 0x7f049013f710 2024-03-25T14:34:54.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.168+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0480001b40 con 0x7f049013f710 2024-03-25T14:34:54.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.168+0000 7f0485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:54.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.168+0000 7f0485ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:34:54.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.168+0000 7f0485ffb640 10 monclient(hunting): dump: 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:54.174 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.169+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 915165165 0 0) 0x7f04900aba00 con 0x7f04901430c0 2024-03-25T14:34:54.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.170+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04901d7490 con 0x7f04901430c0 2024-03-25T14:34:54.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.170+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0488001bd0 con 0x7f04901430c0 2024-03-25T14:34:54.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.170+0000 7f0485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:54.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.170+0000 7f0485ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:54.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.170+0000 7f0485ffb640 10 monclient(hunting): dump: 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:54.176 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.179+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3429458487 0 0) 0x7f04901d7490 con 0x7f04901430c0 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 >> v1:172.21.15.121:6789/0 conn(0x7f04900a98f0 legacy=0x7f04901386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 >> v1:172.21.15.26:6789/0 conn(0x7f049013f710 legacy=0x7f04901d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 1 monclient: found mon.c 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04900abc30 con 0x7f04901430c0 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 10 monclient: _finish_auth 0 2024-03-25T14:34:54.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0485ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 5 monclient: authenticate success, global_id 4535 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 10 monclient: _renew_subs 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0490137940 con 0x7f04901430c0 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 10 monclient: _renew_subs 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0490135220 con 0x7f04901430c0 2024-03-25T14:34:54.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.180+0000 7f0497433640 1 librados: init done 2024-03-25T14:34:54.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.181+0000 7f0497433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:54.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.181+0000 7f0497433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:54.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.181+0000 7f0497433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:54.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.181+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04900627b0 con 0x7f04901430c0 2024-03-25T14:34:54.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0488001fe0 con 0x7f04901430c0 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0488003b30 con 0x7f04901430c0 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 10 monclient: dump: 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:34:54.517 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:34:54.518 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:34:54.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f04877fe640 4 set_mon_vals no callback set 2024-03-25T14:34:54.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.511+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f0488019b00 con 0x7f04901430c0 2024-03-25T14:34:54.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.512+0000 7f04951cf640 10 monclient: get_auth_request con 0x7f047406e110 auth_method 0 2024-03-25T14:34:54.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.512+0000 7f0485ffb640 10 client.4535.objecter ms_handle_connect 0x7f047406e110 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f04880537c0 con 0x7f04901430c0 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 10 client.4535.objecter ms_dispatch 0x7f0490000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f048804f100 con 0x7f04901430c0 2024-03-25T14:34:54.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:34:54.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.518+0000 7f0485ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:34:54.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.758+0000 7f0497433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:54.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.758+0000 7f0497433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:54.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.758+0000 7f0497433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:34:54.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.758+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04901d7830 con 0x7f04901430c0 2024-03-25T14:34:54.806 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.800+0000 7f0485ffb640 1 -- 172.21.15.26:0/267901716 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f0488051200 con 0x7f04901430c0 2024-03-25T14:34:54.806 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.800+0000 7f0485ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:34:54.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.800+0000 7f0485ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:34:54.807 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:34:54.807 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 10 librados: watch_flush enter 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 10 librados: watch_flush exit 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 10 client.4535.objecter successfully canceled tick 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 >> v1:172.21.15.121:6800/288813066 conn(0x7f047406e110 legacy=0x7f04740705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 10 monclient: shutdown 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 >> v1:172.21.15.26:6790/0 conn(0x7f04901430c0 legacy=0x7f0490133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:34:54.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 shutdown_connections 2024-03-25T14:34:54.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 >> 172.21.15.26:0/267901716 conn(0x7f04900b7ea0 msgr2=0x7f0490074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:34:54.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 shutdown_connections 2024-03-25T14:34:54.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 -- 172.21.15.26:0/267901716 wait complete. 2024-03-25T14:34:54.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:34:54.803+0000 7f0497433640 1 librados: shutdown 2024-03-25T14:34:54.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:01.628 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:01.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.738+0000 7f818a20d640 10 monclient: get_monmap_and_config 2024-03-25T14:35:01.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.738+0000 7f818a20d640 10 monclient: build_initial_monmap 2024-03-25T14:35:01.743 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.738+0000 7f818a20d640 10 monclient: monmap: 2024-03-25T14:35:01.743 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:01.739868+0000 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:01.739868+0000 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:01.744 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:01.745 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.738+0000 7f818a20d640 1 Processor -- start 2024-03-25T14:35:01.745 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.738+0000 7f818a20d640 1 -- start start 2024-03-25T14:35:01.745 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.738+0000 7f818a20d640 10 monclient: init 2024-03-25T14:35:01.745 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient(hunting): picked mon.noname-a con 0x7f81840b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient(hunting): picked mon.noname-c con 0x7f81840b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient(hunting): picked mon.noname-b con 0x7f818414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81840af940 con 0x7f81840b4d60 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81840b0a70 con 0x7f81840b1730 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81840b0d50 con 0x7f818414be90 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.739+0000 7f818a20d640 10 monclient(hunting): authenticate will time out at 1986.843154s 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.740+0000 7f8182ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f81840b1730 0x7f81840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:48806/0 (socket says 172.21.15.26:48806) 2024-03-25T14:35:01.746 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.740+0000 7f8182ffd640 1 -- 172.21.15.26:0/122195269 learned_addr learned my addr 172.21.15.26:0/122195269 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:01.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.802+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316733886 0 0) 0x7f81840af940 con 0x7f81840b4d60 2024-03-25T14:35:01.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.802+0000 7f81827fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:01.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.802+0000 7f81827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:01.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.802+0000 7f81827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:01.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.802+0000 7f81827fc640 10 monclient(hunting): my global_id is 4587 2024-03-25T14:35:01.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.802+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816c003520 con 0x7f81840b4d60 2024-03-25T14:35:01.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.813+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148102861 0 0) 0x7f81840b0d50 con 0x7f818414be90 2024-03-25T14:35:01.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.813+0000 7f81827fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:01.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.813+0000 7f81827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:01.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.813+0000 7f81827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:01.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.813+0000 7f81827fc640 10 monclient(hunting): my global_id is 4538 2024-03-25T14:35:01.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.813+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81840af940 con 0x7f818414be90 2024-03-25T14:35:01.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967051581 0 0) 0x7f81840af940 con 0x7f818414be90 2024-03-25T14:35:01.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 >> v1:172.21.15.121:6789/0 conn(0x7f81840b1730 legacy=0x7f81840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:01.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 >> v1:172.21.15.26:6789/0 conn(0x7f81840b4d60 legacy=0x7f818413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:01.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: _finish_hunting 0 2024-03-25T14:35:01.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 1 monclient: found mon.noname-b 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81840b1030 con 0x7f818414be90 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: _finish_auth 0 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: _check_auth_tickets 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f817c0019f0 con 0x7f818414be90 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f81827fc640 10 monclient: dump: 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:01.837 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f818a20d640 5 monclient: authenticate success, global_id 4538 2024-03-25T14:35:01.838 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.830+0000 7f818a20d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f817c001b50 con 0x7f818414be90 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f81827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f81827fc640 1 -- 172.21.15.26:0/122195269 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f817c001fa0 con 0x7f818414be90 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f81827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f81827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f81827fc640 10 monclient: dump: 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:01.854 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f818a20d640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:01.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.848+0000 7f818a20d640 4 set_mon_vals no callback set 2024-03-25T14:35:01.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 10 monclient: shutdown 2024-03-25T14:35:01.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:01.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 1 -- 172.21.15.26:0/122195269 >> v1:172.21.15.26:6790/0 conn(0x7f818414be90 legacy=0x7f818414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:01.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 1 -- 172.21.15.26:0/122195269 shutdown_connections 2024-03-25T14:35:01.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 1 -- 172.21.15.26:0/122195269 >> 172.21.15.26:0/122195269 conn(0x7f81840617a0 msgr2=0x7f8184061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 1 -- 172.21.15.26:0/122195269 shutdown_connections 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.849+0000 7f818a20d640 1 -- 172.21.15.26:0/122195269 wait complete. 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.850+0000 7f818a20d640 10 monclient: build_initial_monmap 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.850+0000 7f818a20d640 10 monclient: monmap: 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:01.858 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.850+0000 7f818a20d640 1 librados: starting msgr at 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.850+0000 7f818a20d640 1 librados: starting objecter 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 Processor -- start 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 -- start start 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 librados: setting wanted keys 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 librados: calling monclient init 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient: init 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient(hunting): picked mon.noname-a con 0x7f81840b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:01.859 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient(hunting): picked mon.noname-b con 0x7f81840b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:01.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient(hunting): picked mon.noname-c con 0x7f818414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:01.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818413a250 con 0x7f81840b1730 2024-03-25T14:35:01.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818413a530 con 0x7f81840b4d60 2024-03-25T14:35:01.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818413a810 con 0x7f818414be90 2024-03-25T14:35:01.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:01.860 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.851+0000 7f818a20d640 10 monclient(hunting): authenticate will time out at 1986.955517s 2024-03-25T14:35:01.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.855+0000 7f8182ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f81840b4d60 0x7f81841d78b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:48860/0 (socket says 172.21.15.26:48860) 2024-03-25T14:35:01.861 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.855+0000 7f8182ffd640 1 -- 172.21.15.26:0/1676292599 learned_addr learned my addr 172.21.15.26:0/1676292599 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:01.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.859+0000 7f816bfff640 10 client.?.objecter ms_handle_connect 0x7f81840b4d60 2024-03-25T14:35:01.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.859+0000 7f816bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:01.864 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.859+0000 7f816bfff640 10 client.?.objecter ms_handle_connect 0x7f81840b1730 2024-03-25T14:35:01.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.859+0000 7f816bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:01.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.866+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746368472 0 0) 0x7f818413a530 con 0x7f81840b4d60 2024-03-25T14:35:01.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.866+0000 7f816bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:01.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.866+0000 7f816bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:01.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.866+0000 7f816bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:01.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.866+0000 7f816bfff640 10 monclient(hunting): my global_id is 4573 2024-03-25T14:35:01.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.866+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8170004eb0 con 0x7f81840b4d60 2024-03-25T14:35:01.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.874+0000 7f816bfff640 10 client.?.objecter ms_handle_connect 0x7f818414be90 2024-03-25T14:35:01.879 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.874+0000 7f816bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:01.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.875+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745014423 0 0) 0x7f818413a250 con 0x7f81840b1730 2024-03-25T14:35:01.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.875+0000 7f816bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:01.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.875+0000 7f816bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:01.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.875+0000 7f816bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:01.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.875+0000 7f816bfff640 10 monclient(hunting): my global_id is 4590 2024-03-25T14:35:01.881 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.875+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818413a530 con 0x7f81840b1730 2024-03-25T14:35:01.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1316280050 0 0) 0x7f8170004eb0 con 0x7f81840b4d60 2024-03-25T14:35:01.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f818413a250 con 0x7f81840b4d60 2024-03-25T14:35:01.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f81780018b0 con 0x7f81840b4d60 2024-03-25T14:35:01.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:01.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 10 monclient(hunting): dump: 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:01.885 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.878+0000 7f816bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:01.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.893+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161145923 0 0) 0x7f818413a810 con 0x7f818414be90 2024-03-25T14:35:01.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.893+0000 7f816bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:01.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.893+0000 7f816bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:01.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.893+0000 7f816bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:01.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.893+0000 7f816bfff640 10 monclient(hunting): my global_id is 4541 2024-03-25T14:35:01.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.893+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8170004eb0 con 0x7f818414be90 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 881599682 0 0) 0x7f818413a250 con 0x7f81840b4d60 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 >> v1:172.21.15.26:6790/0 conn(0x7f818414be90 legacy=0x7f81841e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 >> v1:172.21.15.26:6789/0 conn(0x7f81840b1730 legacy=0x7f8184134fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 1 monclient: found mon.b 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:01.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818413ac30 con 0x7f81840b4d60 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 10 monclient: _finish_auth 0 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f816bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 5 monclient: authenticate success, global_id 4573 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 10 monclient: _renew_subs 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81841356f0 con 0x7f81840b4d60 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 10 monclient: _renew_subs 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8184143420 con 0x7f81840b4d60 2024-03-25T14:35:01.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.894+0000 7f818a20d640 1 librados: init done 2024-03-25T14:35:01.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.895+0000 7f818a20d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:01.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.895+0000 7f818a20d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:01.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.895+0000 7f818a20d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:01.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:01.895+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81841436c0 con 0x7f81840b4d60 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81780047e0 con 0x7f81840b4d60 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f816bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8178004c30 con 0x7f81840b4d60 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f816bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f816bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f816bfff640 10 monclient: dump: 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:02.172 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:02.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.166+0000 7f8181ffb640 4 set_mon_vals no callback set 2024-03-25T14:35:02.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.167+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f8178004e10 con 0x7f81840b4d60 2024-03-25T14:35:02.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.168+0000 7f81837fe640 10 monclient: get_auth_request con 0x7f817006de00 auth_method 0 2024-03-25T14:35:02.174 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.168+0000 7f816bfff640 10 client.4573.objecter ms_handle_connect 0x7f817006de00 2024-03-25T14:35:02.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.173+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f81780547d0 con 0x7f81840b4d60 2024-03-25T14:35:02.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.173+0000 7f816bfff640 10 client.4573.objecter ms_dispatch 0x7f8184000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:02.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.173+0000 7f816bfff640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:02.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.173+0000 7f816bfff640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:02.179 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.173+0000 7f816bfff640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T14:35:02.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.174+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81780502e0 con 0x7f81840b4d60 2024-03-25T14:35:02.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.174+0000 7f816bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:02.180 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.174+0000 7f816bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:02.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.390+0000 7f818a20d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:02.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.390+0000 7f818a20d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:02.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.390+0000 7f818a20d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:02.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.390+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81840627b0 con 0x7f81840b4d60 2024-03-25T14:35:02.435 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:21.296 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.448 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:21.309 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.492 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:21.353 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.492 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:21.353 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.506 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:21.382 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.508 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:21.369 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.505+0000 7f816bfff640 1 -- 172.21.15.26:0/1676292599 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f8178052210 con 0x7f81840b4d60 2024-03-25T14:35:02.510 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.505+0000 7f816bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:02.511 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.505+0000 7f816bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:02.511 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:02.511 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:02.511 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:21.388 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 10 librados: watch_flush enter 2024-03-25T14:35:02.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 10 librados: watch_flush exit 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 10 client.4573.objecter successfully canceled tick 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 >> v1:172.21.15.121:6800/288813066 conn(0x7f817006de00 legacy=0x7f81700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 10 monclient: shutdown 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 >> v1:172.21.15.121:6789/0 conn(0x7f81840b4d60 legacy=0x7f81841d78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 shutdown_connections 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 >> 172.21.15.26:0/1676292599 conn(0x7f81840617a0 msgr2=0x7f81840b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 shutdown_connections 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 -- 172.21.15.26:0/1676292599 wait complete. 2024-03-25T14:35:02.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:02.507+0000 7f818a20d640 1 librados: shutdown 2024-03-25T14:35:02.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:02.533 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:21.409 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:02.534 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:21.395 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:09.331 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:09.447 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.442+0000 7f802030e640 10 monclient: get_monmap_and_config 2024-03-25T14:35:09.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.442+0000 7f802030e640 10 monclient: build_initial_monmap 2024-03-25T14:35:09.448 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.442+0000 7f802030e640 10 monclient: monmap: 2024-03-25T14:35:09.448 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:09.448 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:09.448 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:09.443178+0000 2024-03-25T14:35:09.448 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:09.443178+0000 2024-03-25T14:35:09.449 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:09.449 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:09.449 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:09.449 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:09.449 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:09.449 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:09.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.442+0000 7f802030e640 1 Processor -- start 2024-03-25T14:35:09.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.443+0000 7f802030e640 1 -- start start 2024-03-25T14:35:09.450 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.443+0000 7f802030e640 10 monclient: init 2024-03-25T14:35:09.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.443+0000 7f802030e640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:09.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.443+0000 7f802030e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:35:09.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.443+0000 7f802030e640 10 monclient(hunting): picked mon.noname-b con 0x7f80180a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:09.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 10 monclient(hunting): picked mon.noname-a con 0x7f801813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:09.452 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 10 monclient(hunting): picked mon.noname-c con 0x7f80181430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:09.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8018142520 con 0x7f801813f710 2024-03-25T14:35:09.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80180b28a0 con 0x7f80181430c0 2024-03-25T14:35:09.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80180b2a30 con 0x7f80180a98f0 2024-03-25T14:35:09.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:09.453 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.444+0000 7f802030e640 10 monclient(hunting): authenticate will time out at 1994.546764s 2024-03-25T14:35:09.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.450+0000 7f801e8ab640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f80181430c0 0x7f801814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60084/0 (socket says 172.21.15.26:60084) 2024-03-25T14:35:09.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.450+0000 7f801e8ab640 1 -- 172.21.15.26:0/2125595785 learned_addr learned my addr 172.21.15.26:0/2125595785 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:09.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.458+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615678903 0 0) 0x7f80180b28a0 con 0x7f80181430c0 2024-03-25T14:35:09.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.458+0000 7f801d0a8640 10 monclient(hunting): _negotiate 2024-03-25T14:35:09.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.458+0000 7f801d0a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:09.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.458+0000 7f801d0a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:09.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.458+0000 7f801d0a8640 10 monclient(hunting): my global_id is 4576 2024-03-25T14:35:09.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.458+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8004003520 con 0x7f80181430c0 2024-03-25T14:35:09.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127917135 0 0) 0x7f8004003520 con 0x7f80181430c0 2024-03-25T14:35:09.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 >> v1:172.21.15.26:6790/0 conn(0x7f80180a98f0 legacy=0x7f80180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:09.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 >> v1:172.21.15.26:6789/0 conn(0x7f801813f710 legacy=0x7f80181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:35:09.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: _finish_hunting 0 2024-03-25T14:35:09.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 1 monclient: found mon.noname-c 2024-03-25T14:35:09.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180b44c0 con 0x7f80181430c0 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: _finish_auth 0 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: _check_auth_tickets 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8010001a10 con 0x7f80181430c0 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f802030e640 5 monclient: authenticate success, global_id 4576 2024-03-25T14:35:09.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f802030e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:09.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:09.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:35:09.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f801d0a8640 10 monclient: dump: 2024-03-25T14:35:09.471 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:09.471 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:09.471 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:09.472 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.464+0000 7f802030e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:09.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8010001ca0 con 0x7f80181430c0 2024-03-25T14:35:09.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f801d0a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:09.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f801d0a8640 1 -- 172.21.15.26:0/2125595785 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f80100020b0 con 0x7f80181430c0 2024-03-25T14:35:09.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f801d0a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:09.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f801d0a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:09.473 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f801d0a8640 10 monclient: dump: 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:09.474 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:09.475 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f802030e640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:09.475 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.468+0000 7f802030e640 4 set_mon_vals no callback set 2024-03-25T14:35:09.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 10 monclient: shutdown 2024-03-25T14:35:09.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:09.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 1 -- 172.21.15.26:0/2125595785 >> v1:172.21.15.121:6789/0 conn(0x7f80181430c0 legacy=0x7f801814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:09.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 1 -- 172.21.15.26:0/2125595785 shutdown_connections 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 1 -- 172.21.15.26:0/2125595785 >> 172.21.15.26:0/2125595785 conn(0x7f80180b7ea0 msgr2=0x7f80180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 1 -- 172.21.15.26:0/2125595785 shutdown_connections 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.469+0000 7f802030e640 1 -- 172.21.15.26:0/2125595785 wait complete. 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.470+0000 7f802030e640 10 monclient: build_initial_monmap 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.470+0000 7f802030e640 10 monclient: monmap: 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:09.479 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.470+0000 7f802030e640 1 librados: starting msgr at 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.470+0000 7f802030e640 1 librados: starting objecter 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.470+0000 7f802030e640 1 Processor -- start 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.470+0000 7f802030e640 1 -- start start 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 1 librados: setting wanted keys 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 1 librados: calling monclient init 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient: init 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient(hunting): picked mon.noname-b con 0x7f80180a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient(hunting): picked mon.noname-a con 0x7f801813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:09.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient(hunting): picked mon.noname-c con 0x7f80181430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:09.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8018070aa0 con 0x7f801813f710 2024-03-25T14:35:09.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801806f810 con 0x7f80180a98f0 2024-03-25T14:35:09.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8018070ec0 con 0x7f80181430c0 2024-03-25T14:35:09.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:09.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.471+0000 7f802030e640 10 monclient(hunting): authenticate will time out at 1994.574203s 2024-03-25T14:35:09.481 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.476+0000 7f801e0aa640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f80180a98f0 0x7f801806f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60088/0 (socket says 172.21.15.26:60088) 2024-03-25T14:35:09.482 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.476+0000 7f801e0aa640 1 -- 172.21.15.26:0/3681889950 learned_addr learned my addr 172.21.15.26:0/3681889950 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:09.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.480+0000 7f800effd640 10 client.?.objecter ms_handle_connect 0x7f80180a98f0 2024-03-25T14:35:09.484 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.480+0000 7f800effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:09.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.487+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857564063 0 0) 0x7f801806f810 con 0x7f80180a98f0 2024-03-25T14:35:09.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.487+0000 7f800effd640 10 monclient(hunting): _negotiate 2024-03-25T14:35:09.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.487+0000 7f800effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:09.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.487+0000 7f800effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:09.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.487+0000 7f800effd640 10 monclient(hunting): my global_id is 4579 2024-03-25T14:35:09.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.487+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ffc004eb0 con 0x7f80180a98f0 2024-03-25T14:35:09.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.494+0000 7f800effd640 10 client.?.objecter ms_handle_connect 0x7f801813f710 2024-03-25T14:35:09.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.494+0000 7f800effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:09.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.495+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163495771 0 0) 0x7f7ffc004eb0 con 0x7f80180a98f0 2024-03-25T14:35:09.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.495+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f801806f810 con 0x7f80180a98f0 2024-03-25T14:35:09.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.496+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8014001b10 con 0x7f80180a98f0 2024-03-25T14:35:09.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.496+0000 7f800effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:09.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.496+0000 7f800effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:09.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.496+0000 7f800effd640 10 monclient(hunting): dump: 2024-03-25T14:35:09.500 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:09.500 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:09.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.496+0000 7f800effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:09.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 610031430 0 0) 0x7f801806f810 con 0x7f80180a98f0 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 >> v1:172.21.15.26:6790/0 conn(0x7f80181430c0 legacy=0x7f8018138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 >> v1:172.21.15.26:6789/0 conn(0x7f801813f710 legacy=0x7f80181dc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 10 monclient: _finish_hunting 0 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 1 monclient: found mon.b 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8018071050 con 0x7f80180a98f0 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 10 monclient: _finish_auth 0 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 10 monclient: _check_auth_tickets 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f800effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 5 monclient: authenticate success, global_id 4579 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 10 monclient: _renew_subs 2024-03-25T14:35:09.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80181dc900 con 0x7f80180a98f0 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 10 monclient: _renew_subs 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f801806e6b0 con 0x7f80180a98f0 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.502+0000 7f802030e640 1 librados: init done 2024-03-25T14:35:09.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.503+0000 7f802030e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:09.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.503+0000 7f802030e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:09.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.503+0000 7f802030e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:09.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.503+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801806f9a0 con 0x7f80180a98f0 2024-03-25T14:35:09.788 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8014001c70 con 0x7f80180a98f0 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f800effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f8014003ae0 con 0x7f80180a98f0 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f800effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f800effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f800effd640 10 monclient: dump: 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:09.789 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.784+0000 7f801c8a7640 4 set_mon_vals no callback set 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.785+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f8014003cc0 con 0x7f80180a98f0 2024-03-25T14:35:09.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.786+0000 7f801d8a9640 10 monclient: get_auth_request con 0x7f7ffc06dc10 auth_method 0 2024-03-25T14:35:09.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.786+0000 7f800effd640 10 client.4579.objecter ms_handle_connect 0x7f7ffc06dc10 2024-03-25T14:35:09.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.793+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f8014054a20 con 0x7f80180a98f0 2024-03-25T14:35:09.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.793+0000 7f800effd640 10 client.4579.objecter ms_dispatch 0x7f8018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:09.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.793+0000 7f800effd640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:09.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.793+0000 7f800effd640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:09.798 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.793+0000 7f800effd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T14:35:09.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.794+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8014050360 con 0x7f80180a98f0 2024-03-25T14:35:09.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.794+0000 7f800effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:09.799 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:09.794+0000 7f800effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:10.049 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.045+0000 7f802030e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:10.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.045+0000 7f802030e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:10.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.045+0000 7f802030e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:10.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.045+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80180627b0 con 0x7f80180a98f0 2024-03-25T14:35:10.066 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.062+0000 7f800effd640 1 -- 172.21.15.26:0/3681889950 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f8014052460 con 0x7f80180a98f0 2024-03-25T14:35:10.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.062+0000 7f800effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:10.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.062+0000 7f800effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:10.067 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:10.067 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 10 librados: watch_flush enter 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 10 librados: watch_flush exit 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 10 client.4579.objecter successfully canceled tick 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 >> v1:172.21.15.121:6800/288813066 conn(0x7f7ffc06dc10 legacy=0x7f7ffc0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 10 monclient: shutdown 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 >> v1:172.21.15.121:6789/0 conn(0x7f80180a98f0 legacy=0x7f801806f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:10.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 shutdown_connections 2024-03-25T14:35:10.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 >> 172.21.15.26:0/3681889950 conn(0x7f80180b7ea0 msgr2=0x7f8018074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:10.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 shutdown_connections 2024-03-25T14:35:10.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 -- 172.21.15.26:0/3681889950 wait complete. 2024-03-25T14:35:10.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:10.065+0000 7f802030e640 1 librados: shutdown 2024-03-25T14:35:10.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:16.887 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.001+0000 7f4c29748640 10 monclient: get_monmap_and_config 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.001+0000 7f4c29748640 10 monclient: build_initial_monmap 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.001+0000 7f4c29748640 10 monclient: monmap: 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:17.002634+0000 2024-03-25T14:35:17.006 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:17.002634+0000 2024-03-25T14:35:17.007 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:17.007 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:17.007 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:17.007 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:17.007 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:17.007 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.002+0000 7f4c29748640 1 Processor -- start 2024-03-25T14:35:17.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.002+0000 7f4c29748640 1 -- start start 2024-03-25T14:35:17.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.002+0000 7f4c29748640 10 monclient: init 2024-03-25T14:35:17.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:17.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:35:17.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient(hunting): picked mon.noname-a con 0x7f4c240a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient(hunting): picked mon.noname-c con 0x7f4c241430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c24142520 con 0x7f4c240a98f0 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b28a0 con 0x7f4c241430c0 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b2a30 con 0x7f4c2413f710 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:17.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.003+0000 7f4c29748640 10 monclient(hunting): authenticate will time out at 2002.106344s 2024-03-25T14:35:17.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.011+0000 7f4c22ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c240a98f0 0x7f4c240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:35644/0 (socket says 172.21.15.26:35644) 2024-03-25T14:35:17.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.011+0000 7f4c22ffd640 1 -- 172.21.15.26:0/1668053956 learned_addr learned my addr 172.21.15.26:0/1668053956 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:17.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.022+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161700505 0 0) 0x7f4c240b28a0 con 0x7f4c241430c0 2024-03-25T14:35:17.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.022+0000 7f4c227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:17.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.022+0000 7f4c227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:17.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.022+0000 7f4c227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:17.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.022+0000 7f4c227fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T14:35:17.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.022+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c10003520 con 0x7f4c241430c0 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2493949719 0 0) 0x7f4c10003520 con 0x7f4c241430c0 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 >> v1:172.21.15.26:6790/0 conn(0x7f4c2413f710 legacy=0x7f4c241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 >> v1:172.21.15.26:6789/0 conn(0x7f4c240a98f0 legacy=0x7f4c240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: _finish_hunting 0 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 1 monclient: found mon.noname-c 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:17.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240b44c0 con 0x7f4c241430c0 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: _finish_auth 0 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: _check_auth_tickets 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c1c001a10 con 0x7f4c241430c0 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c227fc640 10 monclient: dump: 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.034 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:17.035 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:17.035 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:17.035 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:17.035 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:17.035 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c29748640 5 monclient: authenticate success, global_id 4582 2024-03-25T14:35:17.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.028+0000 7f4c29748640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c1c001b70 con 0x7f4c241430c0 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c227fc640 1 -- 172.21.15.26:0/1668053956 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c1c001fc0 con 0x7f4c241430c0 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c29748640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c29748640 4 set_mon_vals no callback set 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.036+0000 7f4c227fc640 10 monclient: dump: 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:17.041 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:17.042 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 10 monclient: shutdown 2024-03-25T14:35:17.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:17.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 1 -- 172.21.15.26:0/1668053956 >> v1:172.21.15.121:6789/0 conn(0x7f4c241430c0 legacy=0x7f4c2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 1 -- 172.21.15.26:0/1668053956 shutdown_connections 2024-03-25T14:35:17.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 1 -- 172.21.15.26:0/1668053956 >> 172.21.15.26:0/1668053956 conn(0x7f4c240b7ea0 msgr2=0x7f4c240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:17.045 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 1 -- 172.21.15.26:0/1668053956 shutdown_connections 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.037+0000 7f4c29748640 1 -- 172.21.15.26:0/1668053956 wait complete. 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.038+0000 7f4c29748640 10 monclient: build_initial_monmap 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.038+0000 7f4c29748640 10 monclient: monmap: 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:17.046 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.038+0000 7f4c29748640 1 librados: starting msgr at 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.038+0000 7f4c29748640 1 librados: starting objecter 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.038+0000 7f4c29748640 1 Processor -- start 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 1 -- start start 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 1 librados: setting wanted keys 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 1 librados: calling monclient init 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient: init 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient(hunting): picked mon.noname-a con 0x7f4c240a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient(hunting): picked mon.noname-c con 0x7f4c241430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c24136ad0 con 0x7f4c240a98f0 2024-03-25T14:35:17.047 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b7520 con 0x7f4c2413f710 2024-03-25T14:35:17.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c24135810 con 0x7f4c241430c0 2024-03-25T14:35:17.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:17.048 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.039+0000 7f4c29748640 10 monclient(hunting): authenticate will time out at 2002.142399s 2024-03-25T14:35:17.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.053+0000 7f4c1bfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4c2413f710 0x7f4c240b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:36662/0 (socket says 172.21.15.26:36662) 2024-03-25T14:35:17.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.053+0000 7f4c1bfff640 1 -- 172.21.15.26:0/753485737 learned_addr learned my addr 172.21.15.26:0/753485737 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:17.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.056+0000 7f4c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f4c2413f710 2024-03-25T14:35:17.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.056+0000 7f4c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:17.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.057+0000 7f4c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f4c240a98f0 2024-03-25T14:35:17.061 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.057+0000 7f4c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:17.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.063+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135994729 0 0) 0x7f4c240b7520 con 0x7f4c2413f710 2024-03-25T14:35:17.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.063+0000 7f4c1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:17.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.063+0000 7f4c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:17.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.063+0000 7f4c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:17.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.063+0000 7f4c1b7fe640 10 monclient(hunting): my global_id is 4585 2024-03-25T14:35:17.067 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.063+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c04004eb0 con 0x7f4c2413f710 2024-03-25T14:35:17.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.069+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144721807 0 0) 0x7f4c24136ad0 con 0x7f4c240a98f0 2024-03-25T14:35:17.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.069+0000 7f4c1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:17.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.069+0000 7f4c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:17.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.069+0000 7f4c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:17.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.069+0000 7f4c1b7fe640 10 monclient(hunting): my global_id is 4599 2024-03-25T14:35:17.073 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.069+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c240b7520 con 0x7f4c240a98f0 2024-03-25T14:35:17.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.070+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4100954936 0 0) 0x7f4c04004eb0 con 0x7f4c2413f710 2024-03-25T14:35:17.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.070+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c24136ad0 con 0x7f4c2413f710 2024-03-25T14:35:17.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.070+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c0c001a30 con 0x7f4c2413f710 2024-03-25T14:35:17.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.070+0000 7f4c1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:17.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.071+0000 7f4c1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:17.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.071+0000 7f4c1b7fe640 10 monclient(hunting): dump: 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:17.076 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.071+0000 7f4c1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:17.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.077+0000 7f4c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f4c241430c0 2024-03-25T14:35:17.081 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.077+0000 7f4c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:17.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.088+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239674190 0 0) 0x7f4c240b7520 con 0x7f4c240a98f0 2024-03-25T14:35:17.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.088+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c04004eb0 con 0x7f4c240a98f0 2024-03-25T14:35:17.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.088+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c140018a0 con 0x7f4c240a98f0 2024-03-25T14:35:17.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.088+0000 7f4c1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:17.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.088+0000 7f4c1b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:35:17.092 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.088+0000 7f4c1b7fe640 10 monclient(hunting): dump: 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:17.093 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.092+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904297202 0 0) 0x7f4c24135810 con 0x7f4c241430c0 2024-03-25T14:35:17.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.092+0000 7f4c1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:17.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.092+0000 7f4c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:17.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.092+0000 7f4c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:17.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.092+0000 7f4c1b7fe640 10 monclient(hunting): my global_id is 4547 2024-03-25T14:35:17.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.092+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c240b7520 con 0x7f4c241430c0 2024-03-25T14:35:17.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 500204971 0 0) 0x7f4c24136ad0 con 0x7f4c2413f710 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 >> v1:172.21.15.26:6790/0 conn(0x7f4c241430c0 legacy=0x7f4c24130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 >> v1:172.21.15.26:6789/0 conn(0x7f4c240a98f0 legacy=0x7f4c24135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 1 monclient: found mon.b 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240b6180 con 0x7f4c2413f710 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c29748640 5 monclient: authenticate success, global_id 4585 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c29748640 10 monclient: _renew_subs 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c29748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:17.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.102+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c240b7a40 con 0x7f4c2413f710 2024-03-25T14:35:17.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:17.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 10 monclient: _renew_subs 2024-03-25T14:35:17.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:17.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c24131840 con 0x7f4c2413f710 2024-03-25T14:35:17.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 1 librados: init done 2024-03-25T14:35:17.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:17.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:17.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.103+0000 7f4c29748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:17.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.104+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c24136f60 con 0x7f4c2413f710 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c0c001c70 con 0x7f4c2413f710 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4c0c0038c0 con 0x7f4c2413f710 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c1b7fe640 10 monclient: dump: 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:17.435 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:17.436 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.430+0000 7f4c21ffb640 4 set_mon_vals no callback set 2024-03-25T14:35:17.437 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.431+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4c0c003aa0 con 0x7f4c2413f710 2024-03-25T14:35:17.437 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.432+0000 7f4c22ffd640 10 monclient: get_auth_request con 0x7f4c0406df00 auth_method 0 2024-03-25T14:35:17.437 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.432+0000 7f4c1b7fe640 10 client.4585.objecter ms_handle_connect 0x7f4c0406df00 2024-03-25T14:35:17.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.435+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f4c0c054710 con 0x7f4c2413f710 2024-03-25T14:35:17.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.435+0000 7f4c1b7fe640 10 client.4585.objecter ms_dispatch 0x7f4c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:17.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.435+0000 7f4c1b7fe640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:17.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.435+0000 7f4c1b7fe640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:17.440 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.436+0000 7f4c1b7fe640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T14:35:17.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.436+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c0c050220 con 0x7f4c2413f710 2024-03-25T14:35:17.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.436+0000 7f4c1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:17.441 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.436+0000 7f4c1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:17.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.686+0000 7f4c29748640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:17.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.686+0000 7f4c29748640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:17.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.686+0000 7f4c29748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:17.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.686+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c240b6310 con 0x7f4c2413f710 2024-03-25T14:35:17.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.751+0000 7f4c1b7fe640 1 -- 172.21.15.26:0/753485737 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f4c0c052150 con 0x7f4c2413f710 2024-03-25T14:35:17.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.751+0000 7f4c1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:17.756 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.751+0000 7f4c1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:17.757 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:17.757 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:17.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 10 librados: watch_flush enter 2024-03-25T14:35:17.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 10 librados: watch_flush exit 2024-03-25T14:35:17.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T14:35:17.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 10 client.4585.objecter successfully canceled tick 2024-03-25T14:35:17.759 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 >> v1:172.21.15.121:6800/288813066 conn(0x7f4c0406df00 legacy=0x7f4c040703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 10 monclient: shutdown 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 >> v1:172.21.15.121:6789/0 conn(0x7f4c2413f710 legacy=0x7f4c240b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 shutdown_connections 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 >> 172.21.15.26:0/753485737 conn(0x7f4c240b7ea0 msgr2=0x7f4c24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 shutdown_connections 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 -- 172.21.15.26:0/753485737 wait complete. 2024-03-25T14:35:17.760 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:17.754+0000 7f4c29748640 1 librados: shutdown 2024-03-25T14:35:17.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:22.265 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:41.126 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.324 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:41.185 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.357 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:08:41.218 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.379 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:41.241 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.384 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:41.246 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.386 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:41.263 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.395 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:41.272 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.421 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:08:41.282 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:22.426 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:08:41.303 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:24.581 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:24.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.680+0000 7f0667e0f640 10 monclient: get_monmap_and_config 2024-03-25T14:35:24.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.680+0000 7f0667e0f640 10 monclient: build_initial_monmap 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.680+0000 7f0667e0f640 10 monclient: monmap: 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:24.681311+0000 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:24.681311+0000 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:24.685 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 1 Processor -- start 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 1 -- start start 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient: init 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f06600b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f06600b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:24.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f066014be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06600af940 con 0x7f06600b1730 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06600b0a70 con 0x7f066014be90 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06600b0d50 con 0x7f06600b4d60 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.681+0000 7f0667e0f640 10 monclient(hunting): authenticate will time out at 2009.784522s 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.682+0000 7f0665bab640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f06600b4d60 0x7f066013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60582/0 (socket says 172.21.15.26:60582) 2024-03-25T14:35:24.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.682+0000 7f0665bab640 1 -- 172.21.15.26:0/3511933242 learned_addr learned my addr 172.21.15.26:0/3511933242 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:24.748 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.744+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025715555 0 0) 0x7f06600af940 con 0x7f06600b1730 2024-03-25T14:35:24.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.744+0000 7f0664ba9640 10 monclient(hunting): _negotiate 2024-03-25T14:35:24.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.744+0000 7f0664ba9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:24.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.744+0000 7f0664ba9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:24.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.744+0000 7f0664ba9640 10 monclient(hunting): my global_id is 4602 2024-03-25T14:35:24.749 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.744+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f064c003520 con 0x7f06600b1730 2024-03-25T14:35:24.763 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899614292 0 0) 0x7f064c003520 con 0x7f06600b1730 2024-03-25T14:35:24.763 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 >> v1:172.21.15.26:6790/0 conn(0x7f06600b4d60 legacy=0x7f066013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:24.763 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 >> v1:172.21.15.121:6789/0 conn(0x7f066014be90 legacy=0x7f066014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: _finish_hunting 0 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 1 monclient: found mon.noname-a 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06600b1030 con 0x7f06600b1730 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: _finish_auth 0 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: _check_auth_tickets 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0650001690 con 0x7f06600b1730 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0667e0f640 5 monclient: authenticate success, global_id 4602 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0667e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0664ba9640 10 monclient: dump: 2024-03-25T14:35:24.764 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:24.765 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.759+0000 7f0667e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06500017f0 con 0x7f06600b1730 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0664ba9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0664ba9640 1 -- 172.21.15.26:0/3511933242 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f0650001c00 con 0x7f06600b1730 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0667e0f640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0667e0f640 4 set_mon_vals no callback set 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0664ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0664ba9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:35:24.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.773+0000 7f0664ba9640 10 monclient: dump: 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:24.779 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:24.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 10 monclient: shutdown 2024-03-25T14:35:24.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:24.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 1 -- 172.21.15.26:0/3511933242 >> v1:172.21.15.26:6789/0 conn(0x7f06600b1730 legacy=0x7f06600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:24.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 1 -- 172.21.15.26:0/3511933242 shutdown_connections 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 1 -- 172.21.15.26:0/3511933242 >> 172.21.15.26:0/3511933242 conn(0x7f06600617a0 msgr2=0x7f0660061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 1 -- 172.21.15.26:0/3511933242 shutdown_connections 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.774+0000 7f0667e0f640 1 -- 172.21.15.26:0/3511933242 wait complete. 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.775+0000 7f0667e0f640 10 monclient: build_initial_monmap 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.775+0000 7f0667e0f640 10 monclient: monmap: 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:24.782 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 1 librados: starting msgr at 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 1 librados: starting objecter 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 1 Processor -- start 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 1 -- start start 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 1 librados: setting wanted keys 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 1 librados: calling monclient init 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 10 monclient: init 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f06600b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f06600b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.776+0000 7f0667e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f066014be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:24.783 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.777+0000 7f0667e0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660143690 con 0x7f066014be90 2024-03-25T14:35:24.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.777+0000 7f0667e0f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066013ee90 con 0x7f06600b1730 2024-03-25T14:35:24.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.777+0000 7f0667e0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066013f020 con 0x7f06600b4d60 2024-03-25T14:35:24.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.777+0000 7f0667e0f640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:24.784 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.777+0000 7f0667e0f640 10 monclient(hunting): authenticate will time out at 2009.879715s 2024-03-25T14:35:24.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.786+0000 7f0665bab640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f06600b1730 0x7f066013b420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:47992/0 (socket says 172.21.15.26:47992) 2024-03-25T14:35:24.790 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.786+0000 7f0665bab640 1 -- 172.21.15.26:0/239520666 learned_addr learned my addr 172.21.15.26:0/239520666 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:24.791 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.787+0000 7f06567fc640 10 client.?.objecter ms_handle_connect 0x7f06600b1730 2024-03-25T14:35:24.792 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.787+0000 7f06567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:24.796 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371357188 0 0) 0x7f066013ee90 con 0x7f06600b1730 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0644004eb0 con 0x7f06600b1730 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 10 client.?.objecter ms_handle_connect 0x7f066014be90 2024-03-25T14:35:24.797 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.792+0000 7f06567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:24.806 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.802+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143362066 0 0) 0x7f0660143690 con 0x7f066014be90 2024-03-25T14:35:24.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.802+0000 7f06567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:24.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.802+0000 7f06567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:24.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.802+0000 7f06567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:24.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.802+0000 7f06567fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T14:35:24.807 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.802+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066013ee90 con 0x7f066014be90 2024-03-25T14:35:24.808 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.804+0000 7f06567fc640 10 client.?.objecter ms_handle_connect 0x7f06600b4d60 2024-03-25T14:35:24.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.804+0000 7f06567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:24.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1349346581 0 0) 0x7f0644004eb0 con 0x7f06600b1730 2024-03-25T14:35:24.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0660143690 con 0x7f06600b1730 2024-03-25T14:35:24.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f065c0018b0 con 0x7f06600b1730 2024-03-25T14:35:24.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:24.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 10 monclient(hunting): dump: 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:24.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.805+0000 7f06567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 786754958 0 0) 0x7f0660143690 con 0x7f06600b1730 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 >> v1:172.21.15.26:6790/0 conn(0x7f06600b4d60 legacy=0x7f06601d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 >> v1:172.21.15.26:6789/0 conn(0x7f066014be90 legacy=0x7f06601d8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 10 monclient: _finish_hunting 0 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 1 monclient: found mon.b 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066013bb30 con 0x7f06600b1730 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 10 monclient: _finish_auth 0 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 10 monclient: _check_auth_tickets 2024-03-25T14:35:24.817 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f06567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 5 monclient: authenticate success, global_id 4591 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 10 monclient: _renew_subs 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066013bd80 con 0x7f06600b1730 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 10 monclient: _renew_subs 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0660143f10 con 0x7f06600b1730 2024-03-25T14:35:24.818 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.812+0000 7f0667e0f640 1 librados: init done 2024-03-25T14:35:24.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.813+0000 7f0667e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:24.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.813+0000 7f0667e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:24.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.813+0000 7f0667e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:24.819 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:24.813+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600b3ca0 con 0x7f06600b1730 2024-03-25T14:35:25.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.029+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f065c0047e0 con 0x7f06600b1730 2024-03-25T14:35:25.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.029+0000 7f06567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:25.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.029+0000 7f0657fff640 4 set_mon_vals no callback set 2024-03-25T14:35:25.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.085+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f065c004c30 con 0x7f06600b1730 2024-03-25T14:35:25.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.085+0000 7f06567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:25.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.085+0000 7f06567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:25.089 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.085+0000 7f06567fc640 10 monclient: dump: 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:25.090 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:25.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.086+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f065c004e10 con 0x7f06600b1730 2024-03-25T14:35:25.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.087+0000 7f06653aa640 10 monclient: get_auth_request con 0x7f064406dcc0 auth_method 0 2024-03-25T14:35:25.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.087+0000 7f06567fc640 10 client.4591.objecter ms_handle_connect 0x7f064406dcc0 2024-03-25T14:35:25.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.088+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f065c054850 con 0x7f06600b1730 2024-03-25T14:35:25.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.088+0000 7f06567fc640 10 client.4591.objecter ms_dispatch 0x7f0660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:25.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.088+0000 7f06567fc640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:25.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.088+0000 7f06567fc640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:25.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.088+0000 7f06567fc640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T14:35:25.095 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.091+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f065c050360 con 0x7f06600b1730 2024-03-25T14:35:25.095 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.091+0000 7f06567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:25.095 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.091+0000 7f06567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:25.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.341+0000 7f0667e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:25.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.341+0000 7f0667e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:25.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.341+0000 7f0667e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:25.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.341+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06600627b0 con 0x7f06600b1730 2024-03-25T14:35:25.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.376+0000 7f06567fc640 1 -- 172.21.15.26:0/239520666 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f065c052290 con 0x7f06600b1730 2024-03-25T14:35:25.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.376+0000 7f06567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:25.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.376+0000 7f06567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:25.381 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:25.382 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:25.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 10 librados: watch_flush enter 2024-03-25T14:35:25.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 10 librados: watch_flush exit 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 10 client.4591.objecter successfully canceled tick 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 >> v1:172.21.15.121:6800/288813066 conn(0x7f064406dcc0 legacy=0x7f0644070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 10 monclient: shutdown 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 >> v1:172.21.15.121:6789/0 conn(0x7f06600b1730 legacy=0x7f066013b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 shutdown_connections 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 >> 172.21.15.26:0/239520666 conn(0x7f06600617a0 msgr2=0x7f066013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 shutdown_connections 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 -- 172.21.15.26:0/239520666 wait complete. 2024-03-25T14:35:25.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:25.379+0000 7f0667e0f640 1 librados: shutdown 2024-03-25T14:35:25.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:32.202 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:32.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.316+0000 7fa085e8d640 10 monclient: get_monmap_and_config 2024-03-25T14:35:32.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.316+0000 7fa085e8d640 10 monclient: build_initial_monmap 2024-03-25T14:35:32.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.316+0000 7fa085e8d640 10 monclient: monmap: 2024-03-25T14:35:32.320 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:32.317066+0000 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:32.317066+0000 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:32.321 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 1 Processor -- start 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 1 -- start start 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 10 monclient: init 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 10 monclient(hunting): picked mon.noname-a con 0x7fa08006ab80 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 10 monclient(hunting): picked mon.noname-c con 0x7fa08006af60 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 10 monclient(hunting): picked mon.noname-b con 0x7fa08013f990 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800afa60 con 0x7fa08006ab80 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800afce0 con 0x7fa08006af60 2024-03-25T14:35:32.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.317+0000 7fa085e8d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800b1150 con 0x7fa08013f990 2024-03-25T14:35:32.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.318+0000 7fa085e8d640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:32.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.318+0000 7fa085e8d640 10 monclient(hunting): authenticate will time out at 2017.420635s 2024-03-25T14:35:32.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.319+0000 7fa07effd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa08006af60 0x7fa0801458c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:56632/0 (socket says 172.21.15.26:56632) 2024-03-25T14:35:32.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.319+0000 7fa07effd640 1 -- 172.21.15.26:0/3834621052 learned_addr learned my addr 172.21.15.26:0/3834621052 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:32.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.329+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521698435 0 0) 0x7fa0800afce0 con 0x7fa08006af60 2024-03-25T14:35:32.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.329+0000 7fa07e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:32.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.329+0000 7fa07e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:32.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.329+0000 7fa07e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:32.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.329+0000 7fa07e7fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T14:35:32.334 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.329+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa068003520 con 0x7fa08006af60 2024-03-25T14:35:32.340 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58215891 0 0) 0x7fa068003520 con 0x7fa08006af60 2024-03-25T14:35:32.340 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 >> v1:172.21.15.26:6790/0 conn(0x7fa08013f990 legacy=0x7fa08014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:35:32.340 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 >> v1:172.21.15.26:6789/0 conn(0x7fa08006ab80 legacy=0x7fa0800724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 1 monclient: found mon.noname-c 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0800aff20 con 0x7fa08006af60 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa085e8d640 5 monclient: authenticate success, global_id 4594 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa085e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:32.341 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa074001690 con 0x7fa08006af60 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa07e7fc640 10 monclient: dump: 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:32.342 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:32.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.336+0000 7fa085e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:32.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa074001920 con 0x7fa08006af60 2024-03-25T14:35:32.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa07e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:32.345 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa07e7fc640 1 -- 172.21.15.26:0/3834621052 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa074001690 con 0x7fa08006af60 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa085e8d640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa085e8d640 4 set_mon_vals no callback set 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa07e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa07e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.341+0000 7fa07e7fc640 10 monclient: dump: 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:32.346 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:32.347 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:32.347 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:32.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 10 monclient: shutdown 2024-03-25T14:35:32.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:32.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 1 -- 172.21.15.26:0/3834621052 >> v1:172.21.15.121:6789/0 conn(0x7fa08006af60 legacy=0x7fa0801458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:32.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 1 -- 172.21.15.26:0/3834621052 shutdown_connections 2024-03-25T14:35:32.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 1 -- 172.21.15.26:0/3834621052 >> 172.21.15.26:0/3834621052 conn(0x7fa08006c830 msgr2=0x7fa0800b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:32.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 1 -- 172.21.15.26:0/3834621052 shutdown_connections 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.342+0000 7fa085e8d640 1 -- 172.21.15.26:0/3834621052 wait complete. 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.343+0000 7fa085e8d640 10 monclient: build_initial_monmap 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.343+0000 7fa085e8d640 10 monclient: monmap: 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:32.349 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.343+0000 7fa085e8d640 1 librados: starting msgr at 2024-03-25T14:35:32.350 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.343+0000 7fa085e8d640 1 librados: starting objecter 2024-03-25T14:35:32.350 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.344+0000 7fa085e8d640 1 Processor -- start 2024-03-25T14:35:32.350 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.344+0000 7fa085e8d640 1 -- start start 2024-03-25T14:35:32.350 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.344+0000 7fa085e8d640 1 librados: setting wanted keys 2024-03-25T14:35:32.350 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.344+0000 7fa085e8d640 1 librados: calling monclient init 2024-03-25T14:35:32.350 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.344+0000 7fa085e8d640 10 monclient: init 2024-03-25T14:35:32.351 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:32.351 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:35:32.351 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient(hunting): picked mon.noname-b con 0x7fa08006ab80 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient(hunting): picked mon.noname-a con 0x7fa08013f990 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0801401c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa080140770 con 0x7fa08013f990 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa080140c00 con 0x7fa08006ab80 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa080145ab0 con 0x7fa0801401c0 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:32.352 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.345+0000 7fa085e8d640 10 monclient(hunting): authenticate will time out at 2017.448169s 2024-03-25T14:35:32.356 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.352+0000 7fa07ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa0801401c0 0x7fa080138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50430/0 (socket says 172.21.15.26:50430) 2024-03-25T14:35:32.357 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.352+0000 7fa07ffff640 1 -- 172.21.15.26:0/2631743995 learned_addr learned my addr 172.21.15.26:0/2631743995 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:32.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.353+0000 7fa067fff640 10 client.?.objecter ms_handle_connect 0x7fa0801401c0 2024-03-25T14:35:32.358 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.354+0000 7fa067fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:32.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.361+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203759490 0 0) 0x7fa080145ab0 con 0x7fa0801401c0 2024-03-25T14:35:32.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.361+0000 7fa067fff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:32.365 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.361+0000 7fa067fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:32.366 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.361+0000 7fa067fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:32.366 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.361+0000 7fa067fff640 10 monclient(hunting): my global_id is 4556 2024-03-25T14:35:32.366 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.361+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06c004eb0 con 0x7fa0801401c0 2024-03-25T14:35:32.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.363+0000 7fa067fff640 10 client.?.objecter ms_handle_connect 0x7fa08006ab80 2024-03-25T14:35:32.367 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.363+0000 7fa067fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:32.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.375+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557597152 0 0) 0x7fa080140c00 con 0x7fa08006ab80 2024-03-25T14:35:32.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.375+0000 7fa067fff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:32.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.375+0000 7fa067fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:32.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.375+0000 7fa067fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:32.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.375+0000 7fa067fff640 10 monclient(hunting): my global_id is 4597 2024-03-25T14:35:32.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.375+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa080145ab0 con 0x7fa08006ab80 2024-03-25T14:35:32.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.376+0000 7fa067fff640 10 client.?.objecter ms_handle_connect 0x7fa08013f990 2024-03-25T14:35:32.380 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.376+0000 7fa067fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:32.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002514656 0 0) 0x7fa06c004eb0 con 0x7fa0801401c0 2024-03-25T14:35:32.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa080140c00 con 0x7fa0801401c0 2024-03-25T14:35:32.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa078001ad0 con 0x7fa0801401c0 2024-03-25T14:35:32.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:32.384 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 10 monclient(hunting): dump: 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:32.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.380+0000 7fa067fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 115949029 0 0) 0x7fa080140c00 con 0x7fa0801401c0 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 >> v1:172.21.15.121:6789/0 conn(0x7fa08006ab80 legacy=0x7fa0801da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 >> v1:172.21.15.26:6789/0 conn(0x7fa08013f990 legacy=0x7fa080137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 10 monclient: _finish_hunting 0 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 1 monclient: found mon.c 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:35:32.394 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080138950 con 0x7fa0801401c0 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 10 monclient: _finish_auth 0 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 10 monclient: _check_auth_tickets 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 5 monclient: authenticate success, global_id 4556 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 10 monclient: _renew_subs 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0801455d0 con 0x7fa0801401c0 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 10 monclient: _renew_subs 2024-03-25T14:35:32.395 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:35:32.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0800b4920 con 0x7fa0801401c0 2024-03-25T14:35:32.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.388+0000 7fa085e8d640 1 librados: init done 2024-03-25T14:35:32.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.389+0000 7fa085e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:32.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.389+0000 7fa085e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:32.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.389+0000 7fa085e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:35:32.396 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.389+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa080141f10 con 0x7fa0801401c0 2024-03-25T14:35:32.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.705+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa078001fe0 con 0x7fa0801401c0 2024-03-25T14:35:32.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.705+0000 7fa067fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:32.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.705+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa078004d80 con 0x7fa0801401c0 2024-03-25T14:35:32.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.705+0000 7fa067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:32.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.706+0000 7fa067fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:35:32.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.706+0000 7fa067fff640 10 monclient: dump: 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:32.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.706+0000 7fa07dffb640 4 set_mon_vals no callback set 2024-03-25T14:35:32.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.707+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fa07801ad50 con 0x7fa0801401c0 2024-03-25T14:35:32.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.707+0000 7fa07f7fe640 10 monclient: get_auth_request con 0x7fa06c06dd50 auth_method 0 2024-03-25T14:35:32.712 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.707+0000 7fa067fff640 10 client.4556.objecter ms_handle_connect 0x7fa06c06dd50 2024-03-25T14:35:32.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fa078054a50 con 0x7fa0801401c0 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 10 client.4556.objecter ms_dispatch 0x7fa080000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa078050390 con 0x7fa0801401c0 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:32.719 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.714+0000 7fa067fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:32.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.978+0000 7fa085e8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:32.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.978+0000 7fa085e8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:32.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.978+0000 7fa085e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:35:32.983 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:32.978+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa080140920 con 0x7fa0801401c0 2024-03-25T14:35:33.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.002+0000 7fa067fff640 1 -- 172.21.15.26:0/2631743995 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fa080140920 con 0x7fa0801401c0 2024-03-25T14:35:33.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.002+0000 7fa067fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:33.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.002+0000 7fa067fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:33.007 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:33.007 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:33.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 10 librados: watch_flush enter 2024-03-25T14:35:33.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 10 librados: watch_flush exit 2024-03-25T14:35:33.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T14:35:33.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 10 client.4556.objecter successfully canceled tick 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 >> v1:172.21.15.121:6800/288813066 conn(0x7fa06c06dd50 legacy=0x7fa06c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 10 monclient: shutdown 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 >> v1:172.21.15.26:6790/0 conn(0x7fa0801401c0 legacy=0x7fa080138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 shutdown_connections 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.004+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 >> 172.21.15.26:0/2631743995 conn(0x7fa08006c830 msgr2=0x7fa0800adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.005+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 shutdown_connections 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.005+0000 7fa085e8d640 1 -- 172.21.15.26:0/2631743995 wait complete. 2024-03-25T14:35:33.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:33.005+0000 7fa085e8d640 1 librados: shutdown 2024-03-25T14:35:33.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:39.829 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.921+0000 7f64f964e640 10 monclient: get_monmap_and_config 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.921+0000 7f64f964e640 10 monclient: build_initial_monmap 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.921+0000 7f64f964e640 10 monclient: monmap: 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:39.922678+0000 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:39.922678+0000 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:39.926 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:39.927 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:39.927 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:39.927 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:39.927 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:39.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.922+0000 7f64f964e640 1 Processor -- start 2024-03-25T14:35:39.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.922+0000 7f64f964e640 1 -- start start 2024-03-25T14:35:39.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.922+0000 7f64f964e640 10 monclient: init 2024-03-25T14:35:39.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:39.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:35:39.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient(hunting): picked mon.noname-b con 0x7f64f40a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient(hunting): picked mon.noname-c con 0x7f64f413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient(hunting): picked mon.noname-a con 0x7f64f41430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f4142520 con 0x7f64f41430c0 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f40b28a0 con 0x7f64f413f710 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f40b2a30 con 0x7f64f40a98f0 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:39.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.923+0000 7f64f964e640 10 monclient(hunting): authenticate will time out at 2025.026098s 2024-03-25T14:35:39.941 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.936+0000 7f64f37fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f64f41430c0 0x7f64f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39664/0 (socket says 172.21.15.26:39664) 2024-03-25T14:35:39.941 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.936+0000 7f64f37fe640 1 -- 172.21.15.26:0/1314905758 learned_addr learned my addr 172.21.15.26:0/1314905758 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:39.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.953+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312185906 0 0) 0x7f64f4142520 con 0x7f64f41430c0 2024-03-25T14:35:39.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.953+0000 7f64f27fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:39.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.953+0000 7f64f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:39.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.953+0000 7f64f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:39.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.953+0000 7f64f27fc640 10 monclient(hunting): my global_id is 4614 2024-03-25T14:35:39.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.953+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e0003520 con 0x7f64f41430c0 2024-03-25T14:35:39.959 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.955+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568893938 0 0) 0x7f64f40b2a30 con 0x7f64f40a98f0 2024-03-25T14:35:39.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.955+0000 7f64f27fc640 10 monclient(hunting): _negotiate 2024-03-25T14:35:39.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.955+0000 7f64f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:39.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.955+0000 7f64f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:39.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.955+0000 7f64f27fc640 10 monclient(hunting): my global_id is 4559 2024-03-25T14:35:39.960 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.955+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f4142520 con 0x7f64f40a98f0 2024-03-25T14:35:39.966 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567334661 0 0) 0x7f64e0003520 con 0x7f64f41430c0 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 >> v1:172.21.15.26:6790/0 conn(0x7f64f40a98f0 legacy=0x7f64f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 >> v1:172.21.15.121:6789/0 conn(0x7f64f413f710 legacy=0x7f64f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: _finish_hunting 0 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 1 monclient: found mon.noname-a 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40b44c0 con 0x7f64f41430c0 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: _finish_auth 0 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: _check_auth_tickets 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64ec001a10 con 0x7f64f41430c0 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f964e640 5 monclient: authenticate success, global_id 4614 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f964e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:39.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f27fc640 10 monclient: dump: 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:39.968 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:39.969 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:39.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.962+0000 7f64f964e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:39.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64ec001b70 con 0x7f64f41430c0 2024-03-25T14:35:39.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:39.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f27fc640 1 -- 172.21.15.26:0/1314905758 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64ec001f80 con 0x7f64f41430c0 2024-03-25T14:35:39.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:39.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:35:39.980 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f27fc640 10 monclient: dump: 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:39.981 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:39.982 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:39.982 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:39.982 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:39.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f964e640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:39.982 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.974+0000 7f64f964e640 4 set_mon_vals no callback set 2024-03-25T14:35:39.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.975+0000 7f64f964e640 10 monclient: shutdown 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.975+0000 7f64f964e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.975+0000 7f64f964e640 1 -- 172.21.15.26:0/1314905758 >> v1:172.21.15.26:6789/0 conn(0x7f64f41430c0 legacy=0x7f64f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.976+0000 7f64f964e640 1 -- 172.21.15.26:0/1314905758 shutdown_connections 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.976+0000 7f64f964e640 1 -- 172.21.15.26:0/1314905758 >> 172.21.15.26:0/1314905758 conn(0x7f64f40b7ea0 msgr2=0x7f64f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.976+0000 7f64f964e640 1 -- 172.21.15.26:0/1314905758 shutdown_connections 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.976+0000 7f64f964e640 1 -- 172.21.15.26:0/1314905758 wait complete. 2024-03-25T14:35:39.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 10 monclient: build_initial_monmap 2024-03-25T14:35:39.987 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 10 monclient: monmap: 2024-03-25T14:35:39.987 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:39.987 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:39.987 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:39.987 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:39.987 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:39.988 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:39.988 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:39.988 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:39.988 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:39.988 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:39.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 1 librados: starting msgr at 2024-03-25T14:35:39.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 1 librados: starting objecter 2024-03-25T14:35:39.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 1 Processor -- start 2024-03-25T14:35:39.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 1 -- start start 2024-03-25T14:35:39.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 1 librados: setting wanted keys 2024-03-25T14:35:39.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 1 librados: calling monclient init 2024-03-25T14:35:39.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.977+0000 7f64f964e640 10 monclient: init 2024-03-25T14:35:39.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:39.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:35:39.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient(hunting): picked mon.noname-a con 0x7f64f40a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:39.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient(hunting): picked mon.noname-b con 0x7f64f413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:39.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient(hunting): picked mon.noname-c con 0x7f64f41430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:39.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f4137d10 con 0x7f64f40a98f0 2024-03-25T14:35:39.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f4136bd0 con 0x7f64f413f710 2024-03-25T14:35:39.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f4136d60 con 0x7f64f41430c0 2024-03-25T14:35:39.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:39.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.978+0000 7f64f964e640 10 monclient(hunting): authenticate will time out at 2025.081091s 2024-03-25T14:35:39.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.989+0000 7f64f2ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f64f40a98f0 0x7f64f4139200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39706/0 (socket says 172.21.15.26:39706) 2024-03-25T14:35:39.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.989+0000 7f64f2ffd640 1 -- 172.21.15.26:0/3474705532 learned_addr learned my addr 172.21.15.26:0/3474705532 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:39.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.990+0000 7f64eb7fe640 10 client.?.objecter ms_handle_connect 0x7f64f413f710 2024-03-25T14:35:39.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.990+0000 7f64eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:39.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.991+0000 7f64eb7fe640 10 client.?.objecter ms_handle_connect 0x7f64f40a98f0 2024-03-25T14:35:39.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.991+0000 7f64eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:40.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71029631 0 0) 0x7f64f4137d10 con 0x7f64f40a98f0 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 10 monclient(hunting): my global_id is 4617 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d4004eb0 con 0x7f64f40a98f0 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 10 client.?.objecter ms_handle_connect 0x7f64f41430c0 2024-03-25T14:35:40.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:39.997+0000 7f64eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:40.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.004+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448527165 0 0) 0x7f64f4136d60 con 0x7f64f41430c0 2024-03-25T14:35:40.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.004+0000 7f64eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:40.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.004+0000 7f64eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:40.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.004+0000 7f64eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:40.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.004+0000 7f64eb7fe640 10 monclient(hunting): my global_id is 4562 2024-03-25T14:35:40.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.004+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f4137d10 con 0x7f64f41430c0 2024-03-25T14:35:40.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.006+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310522817 0 0) 0x7f64f4136bd0 con 0x7f64f413f710 2024-03-25T14:35:40.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.006+0000 7f64eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:40.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.006+0000 7f64eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:40.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.006+0000 7f64eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:40.010 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.006+0000 7f64eb7fe640 10 monclient(hunting): my global_id is 4600 2024-03-25T14:35:40.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.006+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f4136d60 con 0x7f64f413f710 2024-03-25T14:35:40.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842155249 0 0) 0x7f64f4137d10 con 0x7f64f41430c0 2024-03-25T14:35:40.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f4136bd0 con 0x7f64f41430c0 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64ec001e80 con 0x7f64f41430c0 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 10 monclient(hunting): dump: 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:40.015 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:40.016 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.010+0000 7f64eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:40.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.014+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990799261 0 0) 0x7f64d4004eb0 con 0x7f64f40a98f0 2024-03-25T14:35:40.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.014+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f4137d10 con 0x7f64f40a98f0 2024-03-25T14:35:40.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.014+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64e4001a10 con 0x7f64f40a98f0 2024-03-25T14:35:40.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.014+0000 7f64eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:40.019 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.014+0000 7f64eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:35:40.020 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.014+0000 7f64eb7fe640 10 monclient(hunting): dump: 2024-03-25T14:35:40.020 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:40.020 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:40.020 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:40.020 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:40.020 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:40.021 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:40.021 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:40.021 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:40.021 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:40.021 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:40.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2835325134 0 0) 0x7f64f4137d10 con 0x7f64f40a98f0 2024-03-25T14:35:40.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 >> v1:172.21.15.26:6790/0 conn(0x7f64f41430c0 legacy=0x7f64f4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 >> v1:172.21.15.121:6789/0 conn(0x7f64f413f710 legacy=0x7f64f4137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 1 monclient: found mon.a 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4133710 con 0x7f64f40a98f0 2024-03-25T14:35:40.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 10 monclient: _finish_auth 0 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 5 monclient: authenticate success, global_id 4617 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 10 monclient: _renew_subs 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f4133960 con 0x7f64f40a98f0 2024-03-25T14:35:40.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:40.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 10 monclient: _renew_subs 2024-03-25T14:35:40.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:35:40.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f413c340 con 0x7f64f40a98f0 2024-03-25T14:35:40.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.021+0000 7f64f964e640 1 librados: init done 2024-03-25T14:35:40.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.022+0000 7f64f964e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:40.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.022+0000 7f64f964e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:40.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.022+0000 7f64f964e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:35:40.030 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.022+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f40627b0 con 0x7f64f40a98f0 2024-03-25T14:35:40.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64e4001c50 con 0x7f64f40a98f0 2024-03-25T14:35:40.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:40.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f64e4003a60 con 0x7f64f40a98f0 2024-03-25T14:35:40.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:40.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:35:40.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64eb7fe640 10 monclient: dump: 2024-03-25T14:35:40.345 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:40.345 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:40.345 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:40.345 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:40.345 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:40.346 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:40.346 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:40.346 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:40.346 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:40.346 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:40.347 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.339+0000 7f64f1ffb640 4 set_mon_vals no callback set 2024-03-25T14:35:40.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.340+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f64e4003c80 con 0x7f64f40a98f0 2024-03-25T14:35:40.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.340+0000 7f64ebfff640 10 monclient: get_auth_request con 0x7f64d406dfb0 auth_method 0 2024-03-25T14:35:40.348 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.341+0000 7f64eb7fe640 10 client.4617.objecter ms_handle_connect 0x7f64d406dfb0 2024-03-25T14:35:40.354 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.349+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f64e4054890 con 0x7f64f40a98f0 2024-03-25T14:35:40.354 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.349+0000 7f64eb7fe640 10 client.4617.objecter ms_dispatch 0x7f64f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:40.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.349+0000 7f64eb7fe640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:40.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.349+0000 7f64eb7fe640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:40.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.349+0000 7f64eb7fe640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T14:35:40.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.349+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64e40503a0 con 0x7f64f40a98f0 2024-03-25T14:35:40.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.350+0000 7f64eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:40.355 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.350+0000 7f64eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:40.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.608+0000 7f64f964e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:40.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.608+0000 7f64f964e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:40.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.608+0000 7f64f964e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:35:40.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.608+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64f4074cc0 con 0x7f64f40a98f0 2024-03-25T14:35:40.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.651+0000 7f64eb7fe640 1 -- 172.21.15.26:0/3474705532 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f64e40522d0 con 0x7f64f40a98f0 2024-03-25T14:35:40.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.651+0000 7f64eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:40.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.651+0000 7f64eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:40.655 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:40.656 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:40.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 10 librados: watch_flush enter 2024-03-25T14:35:40.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 10 librados: watch_flush exit 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 10 client.4617.objecter successfully canceled tick 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 >> v1:172.21.15.121:6800/288813066 conn(0x7f64d406dfb0 legacy=0x7f64d4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 10 monclient: shutdown 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 >> v1:172.21.15.26:6789/0 conn(0x7f64f40a98f0 legacy=0x7f64f4139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 shutdown_connections 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 >> 172.21.15.26:0/3474705532 conn(0x7f64f40b7ea0 msgr2=0x7f64f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 shutdown_connections 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 -- 172.21.15.26:0/3474705532 wait complete. 2024-03-25T14:35:40.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:40.653+0000 7f64f964e640 1 librados: shutdown 2024-03-25T14:35:40.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:42.331 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:01.192 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.385 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:01.246 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.430 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:01.291 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.443 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:01.320 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.443 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:01.305 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.452 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:01.329 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.455 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:01.317 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.484 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:01.345 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:42.490 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:01.367 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:35:47.475 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.587+0000 7f5ea1a0f640 10 monclient: get_monmap_and_config 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.587+0000 7f5ea1a0f640 10 monclient: build_initial_monmap 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.588+0000 7f5ea1a0f640 10 monclient: monmap: 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:47.588859+0000 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:47.588859+0000 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:47.592 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:47.593 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:47.593 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:47.593 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:47.593 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.588+0000 7f5ea1a0f640 1 Processor -- start 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.588+0000 7f5ea1a0f640 1 -- start start 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.588+0000 7f5ea1a0f640 10 monclient: init 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f5e9c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:47.594 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f5e9c13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:47.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f5e9c1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:47.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c142520 con 0x7f5e9c13f710 2024-03-25T14:35:47.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0b28a0 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0b2a30 con 0x7f5e9c1430c0 2024-03-25T14:35:47.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:47.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.589+0000 7f5ea1a0f640 10 monclient(hunting): authenticate will time out at 2032.692576s 2024-03-25T14:35:47.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.605+0000 7f5e9affd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5e9c0a98f0 0x7f5e9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:34260/0 (socket says 172.21.15.26:34260) 2024-03-25T14:35:47.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.605+0000 7f5e9affd640 1 -- 172.21.15.26:0/3389915488 learned_addr learned my addr 172.21.15.26:0/3389915488 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:47.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.623+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891148300 0 0) 0x7f5e9c0b28a0 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.623+0000 7f5e99ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:35:47.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.623+0000 7f5e99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:47.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.623+0000 7f5e99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:47.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.623+0000 7f5e99ffb640 10 monclient(hunting): my global_id is 4603 2024-03-25T14:35:47.627 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.623+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84003520 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.630+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062687172 0 0) 0x7f5e9c0b2a30 con 0x7f5e9c1430c0 2024-03-25T14:35:47.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.630+0000 7f5e99ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:35:47.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.630+0000 7f5e99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:47.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.630+0000 7f5e99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:47.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.630+0000 7f5e99ffb640 10 monclient(hunting): my global_id is 4565 2024-03-25T14:35:47.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.630+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c0b28a0 con 0x7f5e9c1430c0 2024-03-25T14:35:47.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1968201440 0 0) 0x7f5e9c0b28a0 con 0x7f5e9c1430c0 2024-03-25T14:35:47.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 >> v1:172.21.15.121:6789/0 conn(0x7f5e9c0a98f0 legacy=0x7f5e9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:47.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 >> v1:172.21.15.26:6789/0 conn(0x7f5e9c13f710 legacy=0x7f5e9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:47.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:35:47.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 1 monclient: found mon.noname-b 2024-03-25T14:35:47.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0b44c0 con 0x7f5e9c1430c0 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: _finish_auth 0 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5e940019f0 con 0x7f5e9c1430c0 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5ea1a0f640 5 monclient: authenticate success, global_id 4565 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5ea1a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5e99ffb640 10 monclient: dump: 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:47.649 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:47.650 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.643+0000 7f5ea1a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:47.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e94001b50 con 0x7f5e9c1430c0 2024-03-25T14:35:47.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5e99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:47.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5e99ffb640 1 -- 172.21.15.26:0/3389915488 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5e94001f20 con 0x7f5e9c1430c0 2024-03-25T14:35:47.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5ea1a0f640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:47.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5ea1a0f640 4 set_mon_vals no callback set 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5e99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.654+0000 7f5e99ffb640 10 monclient: dump: 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:47.659 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:47.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 10 monclient: shutdown 2024-03-25T14:35:47.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:47.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/3389915488 >> v1:172.21.15.26:6790/0 conn(0x7f5e9c1430c0 legacy=0x7f5e9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:47.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/3389915488 shutdown_connections 2024-03-25T14:35:47.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/3389915488 >> 172.21.15.26:0/3389915488 conn(0x7f5e9c0b7ea0 msgr2=0x7f5e9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:47.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/3389915488 shutdown_connections 2024-03-25T14:35:47.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.655+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/3389915488 wait complete. 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 10 monclient: build_initial_monmap 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 10 monclient: monmap: 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:47.663 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 1 librados: starting msgr at 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 1 librados: starting objecter 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 1 Processor -- start 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 1 -- start start 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 1 librados: setting wanted keys 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 1 librados: calling monclient init 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.656+0000 7f5ea1a0f640 10 monclient: init 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f5e9c0a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f5e9c13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:47.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f5e9c1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:47.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c139590 con 0x7f5e9c1430c0 2024-03-25T14:35:47.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c139800 con 0x7f5e9c13f710 2024-03-25T14:35:47.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c1389f0 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:47.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.657+0000 7f5ea1a0f640 10 monclient(hunting): authenticate will time out at 2032.760309s 2024-03-25T14:35:47.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.676+0000 7f5e9affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5e9c0a98f0 0x7f5e9c1376f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:53136/0 (socket says 172.21.15.26:53136) 2024-03-25T14:35:47.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.676+0000 7f5e9affd640 1 -- 172.21.15.26:0/4226505680 learned_addr learned my addr 172.21.15.26:0/4226505680 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:47.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.678+0000 7f5e83fff640 10 client.?.objecter ms_handle_connect 0x7f5e9c13f710 2024-03-25T14:35:47.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.678+0000 7f5e83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:47.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.679+0000 7f5e83fff640 10 client.?.objecter ms_handle_connect 0x7f5e9c0a98f0 2024-03-25T14:35:47.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.679+0000 7f5e83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:47.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.687+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828912390 0 0) 0x7f5e9c139800 con 0x7f5e9c13f710 2024-03-25T14:35:47.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.687+0000 7f5e83fff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:47.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.687+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:47.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.687+0000 7f5e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:47.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.687+0000 7f5e83fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T14:35:47.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.687+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e88004eb0 con 0x7f5e9c13f710 2024-03-25T14:35:47.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.689+0000 7f5e83fff640 10 client.?.objecter ms_handle_connect 0x7f5e9c1430c0 2024-03-25T14:35:47.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.689+0000 7f5e83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:47.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.695+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665845739 0 0) 0x7f5e9c1389f0 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.695+0000 7f5e83fff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:47.699 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.695+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:47.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.695+0000 7f5e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:47.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.695+0000 7f5e83fff640 10 monclient(hunting): my global_id is 4568 2024-03-25T14:35:47.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.695+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c139800 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.707+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425441024 0 0) 0x7f5e9c139590 con 0x7f5e9c1430c0 2024-03-25T14:35:47.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.707+0000 7f5e83fff640 10 monclient(hunting): _negotiate 2024-03-25T14:35:47.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.707+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:47.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.707+0000 7f5e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:47.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.707+0000 7f5e83fff640 10 monclient(hunting): my global_id is 4623 2024-03-25T14:35:47.711 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.707+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c1389f0 con 0x7f5e9c1430c0 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2217900341 0 0) 0x7f5e88004eb0 con 0x7f5e9c13f710 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e9c139590 con 0x7f5e9c13f710 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5e900018c0 con 0x7f5e9c13f710 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 10 monclient(hunting): dump: 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:47.721 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:47.722 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.717+0000 7f5e83fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.720+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430718997 0 0) 0x7f5e9c139800 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.720+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e88004eb0 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.720+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5e8c001990 con 0x7f5e9c0a98f0 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.720+0000 7f5e83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.721+0000 7f5e83fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.721+0000 7f5e83fff640 10 monclient(hunting): dump: 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:47.725 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:47.726 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:47.726 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:47.726 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:47.726 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:47.726 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4110930735 0 0) 0x7f5e9c139590 con 0x7f5e9c13f710 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 >> v1:172.21.15.26:6790/0 conn(0x7f5e9c0a98f0 legacy=0x7f5e9c1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 >> v1:172.21.15.26:6789/0 conn(0x7f5e9c1430c0 legacy=0x7f5e9c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 10 monclient: _finish_hunting 0 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 1 monclient: found mon.b 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c137f20 con 0x7f5e9c13f710 2024-03-25T14:35:47.728 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 10 monclient: _finish_auth 0 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 10 monclient: _check_auth_tickets 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5e83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 5 monclient: authenticate success, global_id 4606 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 10 monclient: _renew_subs 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c1381d0 con 0x7f5e9c13f710 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 10 monclient: _renew_subs 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c1d7be0 con 0x7f5e9c13f710 2024-03-25T14:35:47.729 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.723+0000 7f5ea1a0f640 1 librados: init done 2024-03-25T14:35:47.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.724+0000 7f5ea1a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:47.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.724+0000 7f5ea1a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:47.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.724+0000 7f5ea1a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:47.730 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:47.724+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c0627b0 con 0x7f5e9c13f710 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e900035f0 con 0x7f5e9c13f710 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5e90003900 con 0x7f5e9c13f710 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e83fff640 10 monclient: dump: 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:48.006 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:48.007 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.001+0000 7f5e997fa640 4 set_mon_vals no callback set 2024-03-25T14:35:48.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.002+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f5e900198d0 con 0x7f5e9c13f710 2024-03-25T14:35:48.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.003+0000 7f5e9affd640 10 monclient: get_auth_request con 0x7f5e8806df20 auth_method 0 2024-03-25T14:35:48.008 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.003+0000 7f5e83fff640 10 client.4606.objecter ms_handle_connect 0x7f5e8806df20 2024-03-25T14:35:48.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f5e900535c0 con 0x7f5e9c13f710 2024-03-25T14:35:48.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 10 client.4606.objecter ms_dispatch 0x7f5e9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:48.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:48.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:48.011 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T14:35:48.012 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e9004f0d0 con 0x7f5e9c13f710 2024-03-25T14:35:48.012 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:48.012 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.007+0000 7f5e83fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:48.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.242+0000 7f5ea1a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:48.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.242+0000 7f5ea1a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:48.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.242+0000 7f5ea1a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:48.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.242+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e9c074cc0 con 0x7f5e9c13f710 2024-03-25T14:35:48.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.281+0000 7f5e83fff640 1 -- 172.21.15.26:0/4226505680 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f5e90051000 con 0x7f5e9c13f710 2024-03-25T14:35:48.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.281+0000 7f5e83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:48.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.281+0000 7f5e83fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:48.286 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:48.286 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:48.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 10 librados: watch_flush enter 2024-03-25T14:35:48.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 10 librados: watch_flush exit 2024-03-25T14:35:48.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T14:35:48.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 10 client.4606.objecter successfully canceled tick 2024-03-25T14:35:48.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 >> v1:172.21.15.121:6800/288813066 conn(0x7f5e8806df20 legacy=0x7f5e880703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:48.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 10 monclient: shutdown 2024-03-25T14:35:48.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:48.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 >> v1:172.21.15.121:6789/0 conn(0x7f5e9c13f710 legacy=0x7f5e9c1d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:48.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 shutdown_connections 2024-03-25T14:35:48.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.283+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 >> 172.21.15.26:0/4226505680 conn(0x7f5e9c0b7ea0 msgr2=0x7f5e9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:48.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.284+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 shutdown_connections 2024-03-25T14:35:48.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.284+0000 7f5ea1a0f640 1 -- 172.21.15.26:0/4226505680 wait complete. 2024-03-25T14:35:48.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:48.284+0000 7f5ea1a0f640 1 librados: shutdown 2024-03-25T14:35:48.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:35:55.107 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:35:55.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.223+0000 7f9faee57640 10 monclient: get_monmap_and_config 2024-03-25T14:35:55.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.223+0000 7f9faee57640 10 monclient: build_initial_monmap 2024-03-25T14:35:55.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.223+0000 7f9faee57640 10 monclient: monmap: 2024-03-25T14:35:55.227 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:35:55.223876+0000 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:35:55.223876+0000 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:35:55.228 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:55.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.223+0000 7f9faee57640 1 Processor -- start 2024-03-25T14:35:55.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 1 -- start start 2024-03-25T14:35:55.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient: init 2024-03-25T14:35:55.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa80a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa81430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8142520 con 0x7f9fa81430c0 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80b28a0 con 0x7f9fa80a98f0 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80b2a30 con 0x7f9fa813f710 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:55.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.224+0000 7f9faee57640 10 monclient(hunting): authenticate will time out at 2040.327328s 2024-03-25T14:35:55.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.236+0000 7f9facbf3640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9fa80a98f0 0x7f9fa80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:46876/0 (socket says 172.21.15.26:46876) 2024-03-25T14:35:55.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.236+0000 7f9facbf3640 1 -- 172.21.15.26:0/3510284802 learned_addr learned my addr 172.21.15.26:0/3510284802 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:55.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.247+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802674159 0 0) 0x7f9fa80b28a0 con 0x7f9fa80a98f0 2024-03-25T14:35:55.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.247+0000 7f9fa77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:55.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.247+0000 7f9fa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:55.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.247+0000 7f9fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:55.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.247+0000 7f9fa77fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T14:35:55.252 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.247+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f98003520 con 0x7f9fa80a98f0 2024-03-25T14:35:55.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.259+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925368477 0 0) 0x7f9fa80b2a30 con 0x7f9fa813f710 2024-03-25T14:35:55.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.259+0000 7f9fa77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:55.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.259+0000 7f9fa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:55.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.259+0000 7f9fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:55.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.259+0000 7f9fa77fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T14:35:55.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.259+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa80b28a0 con 0x7f9fa813f710 2024-03-25T14:35:55.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.260+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973635997 0 0) 0x7f9fa8142520 con 0x7f9fa81430c0 2024-03-25T14:35:55.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.260+0000 7f9fa77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:35:55.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.260+0000 7f9fa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:55.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.260+0000 7f9fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:55.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.260+0000 7f9fa77fe640 10 monclient(hunting): my global_id is 4626 2024-03-25T14:35:55.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.260+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa80b2a30 con 0x7f9fa81430c0 2024-03-25T14:35:55.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263172845 0 0) 0x7f9f98003520 con 0x7f9fa80a98f0 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 >> v1:172.21.15.26:6790/0 conn(0x7f9fa813f710 legacy=0x7f9fa81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 >> v1:172.21.15.26:6789/0 conn(0x7f9fa81430c0 legacy=0x7f9fa814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: _finish_hunting 0 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 1 monclient: found mon.noname-c 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80b44c0 con 0x7f9fa80a98f0 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: _finish_auth 0 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: _check_auth_tickets 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:55.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9f9c001690 con 0x7f9fa80a98f0 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9faee57640 5 monclient: authenticate success, global_id 4609 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9faee57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9fa77fe640 10 monclient: dump: 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:55.274 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:55.275 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:55.275 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:55.275 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.268+0000 7f9faee57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:35:55.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f9c0017f0 con 0x7f9fa80a98f0 2024-03-25T14:35:55.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9fa77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9fa77fe640 1 -- 172.21.15.26:0/3510284802 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9f9c001c00 con 0x7f9fa80a98f0 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9fa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9fa77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9fa77fe640 10 monclient: dump: 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:55.279 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:55.280 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:55.280 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:55.280 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:55.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9faee57640 10 monclient: get_monmap_and_config success 2024-03-25T14:35:55.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.274+0000 7f9faee57640 4 set_mon_vals no callback set 2024-03-25T14:35:55.282 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 10 monclient: shutdown 2024-03-25T14:35:55.282 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 1 -- 172.21.15.26:0/3510284802 >> v1:172.21.15.121:6789/0 conn(0x7f9fa80a98f0 legacy=0x7f9fa80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 1 -- 172.21.15.26:0/3510284802 shutdown_connections 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 1 -- 172.21.15.26:0/3510284802 >> 172.21.15.26:0/3510284802 conn(0x7f9fa80b7ea0 msgr2=0x7f9fa80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 1 -- 172.21.15.26:0/3510284802 shutdown_connections 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.275+0000 7f9faee57640 1 -- 172.21.15.26:0/3510284802 wait complete. 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.276+0000 7f9faee57640 10 monclient: build_initial_monmap 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.276+0000 7f9faee57640 10 monclient: monmap: 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:35:55.283 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.276+0000 7f9faee57640 1 librados: starting msgr at 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.276+0000 7f9faee57640 1 librados: starting objecter 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 Processor -- start 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 -- start start 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 librados: setting wanted keys 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 librados: calling monclient init 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient: init 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient: _reopen_session rank -1 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa80a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:35:55.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:35:55.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa81430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:35:55.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8138250 con 0x7f9fa80a98f0 2024-03-25T14:35:55.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8137d50 con 0x7f9fa81430c0 2024-03-25T14:35:55.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8137ee0 con 0x7f9fa813f710 2024-03-25T14:35:55.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient(hunting): _renew_subs 2024-03-25T14:35:55.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.277+0000 7f9faee57640 10 monclient(hunting): authenticate will time out at 2040.380425s 2024-03-25T14:35:55.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.293+0000 7f9facbf3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9fa80a98f0 0x7f9fa8137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53144/0 (socket says 172.21.15.26:53144) 2024-03-25T14:35:55.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.293+0000 7f9facbf3640 1 -- 172.21.15.26:0/3269116858 learned_addr learned my addr 172.21.15.26:0/3269116858 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:35:55.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.296+0000 7f9fa57fa640 10 client.?.objecter ms_handle_connect 0x7f9fa81430c0 2024-03-25T14:35:55.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.296+0000 7f9fa57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:55.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.298+0000 7f9fa57fa640 10 client.?.objecter ms_handle_connect 0x7f9fa80a98f0 2024-03-25T14:35:55.302 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.298+0000 7f9fa57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:55.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.300+0000 7f9fa57fa640 10 client.?.objecter ms_handle_connect 0x7f9fa813f710 2024-03-25T14:35:55.304 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.300+0000 7f9fa57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:35:55.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.314+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148200636 0 0) 0x7f9fa8137ee0 con 0x7f9fa813f710 2024-03-25T14:35:55.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.314+0000 7f9fa57fa640 10 monclient(hunting): _negotiate 2024-03-25T14:35:55.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.314+0000 7f9fa57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:55.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.314+0000 7f9fa57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:55.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.314+0000 7f9fa57fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T14:35:55.318 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.314+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c004eb0 con 0x7f9fa813f710 2024-03-25T14:35:55.320 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.316+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359215027 0 0) 0x7f9fa8137d50 con 0x7f9fa81430c0 2024-03-25T14:35:55.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.316+0000 7f9fa57fa640 10 monclient(hunting): _negotiate 2024-03-25T14:35:55.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.316+0000 7f9fa57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:55.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.316+0000 7f9fa57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:55.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.316+0000 7f9fa57fa640 10 monclient(hunting): my global_id is 4612 2024-03-25T14:35:55.321 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.316+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8137ee0 con 0x7f9fa81430c0 2024-03-25T14:35:55.330 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.326+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479865558 0 0) 0x7f9fa8138250 con 0x7f9fa80a98f0 2024-03-25T14:35:55.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.326+0000 7f9fa57fa640 10 monclient(hunting): _negotiate 2024-03-25T14:35:55.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.326+0000 7f9fa57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:35:55.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.326+0000 7f9fa57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:35:55.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.326+0000 7f9fa57fa640 10 monclient(hunting): my global_id is 4629 2024-03-25T14:35:55.331 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.326+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8137d50 con 0x7f9fa80a98f0 2024-03-25T14:35:55.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3334989154 0 0) 0x7f9fa8137ee0 con 0x7f9fa81430c0 2024-03-25T14:35:55.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa8138250 con 0x7f9fa81430c0 2024-03-25T14:35:55.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9fa0001b50 con 0x7f9fa81430c0 2024-03-25T14:35:55.343 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 10 monclient(hunting): dump: 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:55.344 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.339+0000 7f9fa57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:35:55.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.354+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2276074141 0 0) 0x7f9fa8138250 con 0x7f9fa81430c0 2024-03-25T14:35:55.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.354+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 >> v1:172.21.15.26:6790/0 conn(0x7f9fa813f710 legacy=0x7f9fa81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 >> v1:172.21.15.26:6789/0 conn(0x7f9fa80a98f0 legacy=0x7f9fa8137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 10 monclient: _finish_hunting 0 2024-03-25T14:35:55.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 1 monclient: found mon.b 2024-03-25T14:35:55.359 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80b7730 con 0x7f9fa81430c0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 10 monclient: _finish_auth 0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 10 monclient: _check_auth_tickets 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9fa57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 5 monclient: authenticate success, global_id 4612 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 10 monclient: _renew_subs 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa80b7920 con 0x7f9fa81430c0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 10 monclient: _renew_subs 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa81e33d0 con 0x7f9fa81430c0 2024-03-25T14:35:55.360 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.355+0000 7f9faee57640 1 librados: init done 2024-03-25T14:35:55.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.356+0000 7f9faee57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:55.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.356+0000 7f9faee57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:55.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.356+0000 7f9faee57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:55.361 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.356+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa80627b0 con 0x7f9fa81430c0 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fa0001fe0 con 0x7f9fa81430c0 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9fa0003b30 con 0x7f9fa81430c0 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa57fa640 10 monclient: dump: 2024-03-25T14:35:55.682 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:35:55.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.677+0000 7f9fa6ffd640 4 set_mon_vals no callback set 2024-03-25T14:35:55.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.679+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f9fa0003d50 con 0x7f9fa81430c0 2024-03-25T14:35:55.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.679+0000 7f9facbf3640 10 monclient: get_auth_request con 0x7f9f8c06ddd0 auth_method 0 2024-03-25T14:35:55.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.679+0000 7f9fa57fa640 10 client.4612.objecter ms_handle_connect 0x7f9f8c06ddd0 2024-03-25T14:35:55.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.683+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f9fa00537e0 con 0x7f9fa81430c0 2024-03-25T14:35:55.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.683+0000 7f9fa57fa640 10 client.4612.objecter ms_dispatch 0x7f9fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:35:55.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.683+0000 7f9fa57fa640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:35:55.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.683+0000 7f9fa57fa640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:35:55.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.683+0000 7f9fa57fa640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T14:35:55.689 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.684+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fa004f2f0 con 0x7f9fa81430c0 2024-03-25T14:35:55.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.684+0000 7f9fa57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:35:55.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.684+0000 7f9fa57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:35:55.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.926+0000 7f9faee57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:55.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.926+0000 7f9faee57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:55.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.926+0000 7f9faee57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:35:55.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.926+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fa8138400 con 0x7f9fa81430c0 2024-03-25T14:35:55.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.965+0000 7f9fa57fa640 1 -- 172.21.15.26:0/3269116858 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f9fa8138400 con 0x7f9fa81430c0 2024-03-25T14:35:55.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.965+0000 7f9fa57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:35:55.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.965+0000 7f9fa57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:35:55.970 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:35:55.971 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:35:55.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.967+0000 7f9faee57640 10 librados: watch_flush enter 2024-03-25T14:35:55.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 10 librados: watch_flush exit 2024-03-25T14:35:55.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T14:35:55.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 10 client.4612.objecter successfully canceled tick 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 >> v1:172.21.15.121:6800/288813066 conn(0x7f9f8c06ddd0 legacy=0x7f9f8c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 10 monclient: shutdown 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 >> v1:172.21.15.121:6789/0 conn(0x7f9fa81430c0 legacy=0x7f9fa81d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 shutdown_connections 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 >> 172.21.15.26:0/3269116858 conn(0x7f9fa80b7ea0 msgr2=0x7f9fa8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 shutdown_connections 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 -- 172.21.15.26:0/3269116858 wait complete. 2024-03-25T14:35:55.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:35:55.968+0000 7f9faee57640 1 librados: shutdown 2024-03-25T14:35:55.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:02.470 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:21.331 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.484 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:21.345 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.523 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:21.384 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.531 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:21.408 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.540 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:21.401 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.540 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:21.417 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.546 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:21.407 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.575 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:21.436 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.578 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:21.455 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:02.792 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.903+0000 7f3f4995d640 10 monclient: get_monmap_and_config 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.903+0000 7f3f4995d640 10 monclient: build_initial_monmap 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.903+0000 7f3f4995d640 10 monclient: monmap: 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:02.904878+0000 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:02.904878+0000 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:02.908 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:02.909 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:02.909 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:02.909 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:02.909 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.903+0000 7f3f4995d640 1 Processor -- start 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.903+0000 7f3f4995d640 1 -- start start 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.903+0000 7f3f4995d640 10 monclient: init 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f440a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f441430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f44142520 con 0x7f3f440a98f0 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f440b28a0 con 0x7f3f441430c0 2024-03-25T14:36:02.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f440b2a30 con 0x7f3f4413f710 2024-03-25T14:36:02.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:02.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.904+0000 7f3f4995d640 10 monclient(hunting): authenticate will time out at 2048.008108s 2024-03-25T14:36:02.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.919+0000 7f3f42ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f440a98f0 0x7f3f440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58966/0 (socket says 172.21.15.26:58966) 2024-03-25T14:36:02.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.919+0000 7f3f42ffd640 1 -- 172.21.15.26:0/2924611315 learned_addr learned my addr 172.21.15.26:0/2924611315 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.946+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605991565 0 0) 0x7f3f440b2a30 con 0x7f3f4413f710 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.946+0000 7f3f41ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.946+0000 7f3f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.946+0000 7f3f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.946+0000 7f3f41ffb640 10 monclient(hunting): my global_id is 4577 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.946+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c003520 con 0x7f3f4413f710 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.947+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158497923 0 0) 0x7f3f44142520 con 0x7f3f440a98f0 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.947+0000 7f3f41ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.947+0000 7f3f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:02.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.947+0000 7f3f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:02.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.947+0000 7f3f41ffb640 10 monclient(hunting): my global_id is 4632 2024-03-25T14:36:02.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.947+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f440b2a30 con 0x7f3f440a98f0 2024-03-25T14:36:02.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.949+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799254583 0 0) 0x7f3f440b28a0 con 0x7f3f441430c0 2024-03-25T14:36:02.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.949+0000 7f3f41ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:36:02.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.949+0000 7f3f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:02.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.949+0000 7f3f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:02.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.949+0000 7f3f41ffb640 10 monclient(hunting): my global_id is 4615 2024-03-25T14:36:02.955 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.949+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f44142520 con 0x7f3f441430c0 2024-03-25T14:36:02.966 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2919221830 0 0) 0x7f3f440b2a30 con 0x7f3f440a98f0 2024-03-25T14:36:02.966 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 >> v1:172.21.15.26:6790/0 conn(0x7f3f4413f710 legacy=0x7f3f441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:02.966 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 >> v1:172.21.15.121:6789/0 conn(0x7f3f441430c0 legacy=0x7f3f4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 1 monclient: found mon.noname-a 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f440b44c0 con 0x7f3f440a98f0 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: _finish_auth 0 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3f34001670 con 0x7f3f440a98f0 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f4995d640 5 monclient: authenticate success, global_id 4632 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f4995d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:36:02.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f41ffb640 10 monclient: dump: 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:02.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.961+0000 7f3f4995d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f340017d0 con 0x7f3f440a98f0 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f41ffb640 1 -- 172.21.15.26:0/2924611315 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3f34001be0 con 0x7f3f440a98f0 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f41ffb640 10 monclient: dump: 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:02.985 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:02.986 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 4 set_mon_vals no callback set 2024-03-25T14:36:02.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 10 monclient: shutdown 2024-03-25T14:36:02.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:02.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 1 -- 172.21.15.26:0/2924611315 >> v1:172.21.15.26:6789/0 conn(0x7f3f440a98f0 legacy=0x7f3f440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:02.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 1 -- 172.21.15.26:0/2924611315 shutdown_connections 2024-03-25T14:36:02.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.980+0000 7f3f4995d640 1 -- 172.21.15.26:0/2924611315 >> 172.21.15.26:0/2924611315 conn(0x7f3f440b7ea0 msgr2=0x7f3f440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:02.988 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.981+0000 7f3f4995d640 1 -- 172.21.15.26:0/2924611315 shutdown_connections 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.981+0000 7f3f4995d640 1 -- 172.21.15.26:0/2924611315 wait complete. 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.981+0000 7f3f4995d640 10 monclient: build_initial_monmap 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.981+0000 7f3f4995d640 10 monclient: monmap: 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:02.989 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 1 librados: starting msgr at 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 1 librados: starting objecter 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 1 Processor -- start 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 1 -- start start 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 1 librados: setting wanted keys 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 1 librados: calling monclient init 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.982+0000 7f3f4995d640 10 monclient: init 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f440a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f441430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f441391e0 con 0x7f3f441430c0 2024-03-25T14:36:02.990 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f441d8380 con 0x7f3f4413f710 2024-03-25T14:36:02.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f44139450 con 0x7f3f440a98f0 2024-03-25T14:36:02.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:02.991 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.983+0000 7f3f4995d640 10 monclient(hunting): authenticate will time out at 2048.086984s 2024-03-25T14:36:03.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.998+0000 7f3f437fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3f441430c0 0x7f3f441340a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59014/0 (socket says 172.21.15.26:59014) 2024-03-25T14:36:03.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.998+0000 7f3f437fe640 1 -- 172.21.15.26:0/1356640542 learned_addr learned my addr 172.21.15.26:0/1356640542 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:03.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.998+0000 7f3f2bfff640 10 client.?.objecter ms_handle_connect 0x7f3f441430c0 2024-03-25T14:36:03.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:02.998+0000 7f3f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:03.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.004+0000 7f3f2bfff640 10 client.?.objecter ms_handle_connect 0x7f3f440a98f0 2024-03-25T14:36:03.009 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.004+0000 7f3f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:03.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.009+0000 7f3f2bfff640 10 client.?.objecter ms_handle_connect 0x7f3f4413f710 2024-03-25T14:36:03.014 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.009+0000 7f3f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:03.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.016+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711427537 0 0) 0x7f3f441391e0 con 0x7f3f441430c0 2024-03-25T14:36:03.021 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.016+0000 7f3f2bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:36:03.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.016+0000 7f3f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:03.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.016+0000 7f3f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:03.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.016+0000 7f3f2bfff640 10 monclient(hunting): my global_id is 4635 2024-03-25T14:36:03.022 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.016+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f30004eb0 con 0x7f3f441430c0 2024-03-25T14:36:03.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.021+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493158756 0 0) 0x7f3f44139450 con 0x7f3f440a98f0 2024-03-25T14:36:03.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.021+0000 7f3f2bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:36:03.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.021+0000 7f3f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:03.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.021+0000 7f3f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:03.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.021+0000 7f3f2bfff640 10 monclient(hunting): my global_id is 4580 2024-03-25T14:36:03.027 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.021+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f441391e0 con 0x7f3f440a98f0 2024-03-25T14:36:03.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.031+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291751411 0 0) 0x7f3f441d8380 con 0x7f3f4413f710 2024-03-25T14:36:03.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.031+0000 7f3f2bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:36:03.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.031+0000 7f3f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:03.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.031+0000 7f3f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:03.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.031+0000 7f3f2bfff640 10 monclient(hunting): my global_id is 4618 2024-03-25T14:36:03.036 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.031+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f44139450 con 0x7f3f4413f710 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2750118868 0 0) 0x7f3f30004eb0 con 0x7f3f441430c0 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f441d8380 con 0x7f3f441430c0 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3f3c001c70 con 0x7f3f441430c0 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 10 monclient(hunting): dump: 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:03.039 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:03.040 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.034+0000 7f3f2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828838907 0 0) 0x7f3f441391e0 con 0x7f3f440a98f0 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f30004eb0 con 0x7f3f440a98f0 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3f34001ae0 con 0x7f3f440a98f0 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient(hunting): dump: 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:03.057 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3568550922 0 0) 0x7f3f441d8380 con 0x7f3f441430c0 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 >> v1:172.21.15.26:6790/0 conn(0x7f3f440a98f0 legacy=0x7f3f441d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 >> v1:172.21.15.121:6789/0 conn(0x7f3f4413f710 legacy=0x7f3f44138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 monclient: found mon.a 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:03.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f44139730 con 0x7f3f441430c0 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient: _finish_auth 0 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 5 monclient: authenticate success, global_id 4635 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 10 monclient: _renew_subs 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f441d7750 con 0x7f3f441430c0 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 10 monclient: _renew_subs 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f441347b0 con 0x7f3f441430c0 2024-03-25T14:36:03.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.051+0000 7f3f4995d640 1 librados: init done 2024-03-25T14:36:03.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.052+0000 7f3f4995d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:03.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.052+0000 7f3f4995d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:03.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.052+0000 7f3f4995d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:03.064 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.052+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f44134ab0 con 0x7f3f441430c0 2024-03-25T14:36:03.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f3c003990 con 0x7f3f441430c0 2024-03-25T14:36:03.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:03.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3f3c003d20 con 0x7f3f441430c0 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f2bfff640 10 monclient: dump: 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:03.378 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:03.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.372+0000 7f3f417fa640 4 set_mon_vals no callback set 2024-03-25T14:36:03.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.373+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f3f3c003f40 con 0x7f3f441430c0 2024-03-25T14:36:03.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.373+0000 7f3f42ffd640 10 monclient: get_auth_request con 0x7f3f3006df70 auth_method 0 2024-03-25T14:36:03.379 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.373+0000 7f3f2bfff640 10 client.4635.objecter ms_handle_connect 0x7f3f3006df70 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f3f3c054a90 con 0x7f3f441430c0 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 10 client.4635.objecter ms_dispatch 0x7f3f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f3c0505a0 con 0x7f3f441430c0 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:03.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.380+0000 7f3f2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:03.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.602+0000 7f3f4995d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:03.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.602+0000 7f3f4995d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:03.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.602+0000 7f3f4995d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:03.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.602+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f44074cc0 con 0x7f3f441430c0 2024-03-25T14:36:03.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.656+0000 7f3f2bfff640 1 -- 172.21.15.26:0/1356640542 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f3f3c0524d0 con 0x7f3f441430c0 2024-03-25T14:36:03.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.656+0000 7f3f2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:03.661 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.656+0000 7f3f2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:03.661 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:03.661 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:03.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 10 librados: watch_flush enter 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 10 librados: watch_flush exit 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 10 client.4635.objecter successfully canceled tick 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 >> v1:172.21.15.121:6800/288813066 conn(0x7f3f3006df70 legacy=0x7f3f30070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 10 monclient: shutdown 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 >> v1:172.21.15.26:6789/0 conn(0x7f3f441430c0 legacy=0x7f3f441340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 shutdown_connections 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 >> 172.21.15.26:0/1356640542 conn(0x7f3f440b7ea0 msgr2=0x7f3f4413d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 shutdown_connections 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 -- 172.21.15.26:0/1356640542 wait complete. 2024-03-25T14:36:03.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:03.658+0000 7f3f4995d640 1 librados: shutdown 2024-03-25T14:36:03.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:10.481 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:10.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.591+0000 7ff45008b640 10 monclient: get_monmap_and_config 2024-03-25T14:36:10.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.591+0000 7ff45008b640 10 monclient: build_initial_monmap 2024-03-25T14:36:10.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.591+0000 7ff45008b640 10 monclient: monmap: 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:10.593294+0000 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:10.593294+0000 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:10.597 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:10.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 1 Processor -- start 2024-03-25T14:36:10.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 1 -- start start 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 10 monclient: init 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4480a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 10 monclient(hunting): picked mon.noname-a con 0x7ff44813f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4481430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff448142520 con 0x7ff44813f710 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.592+0000 7ff45008b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480b28a0 con 0x7ff4481430c0 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.593+0000 7ff45008b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480b2a30 con 0x7ff4480a98f0 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.593+0000 7ff45008b640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:10.600 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.593+0000 7ff45008b640 10 monclient(hunting): authenticate will time out at 2055.696617s 2024-03-25T14:36:10.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.604+0000 7ff44de27640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff4480a98f0 0x7ff4480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:44002/0 (socket says 172.21.15.26:44002) 2024-03-25T14:36:10.609 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.604+0000 7ff44de27640 1 -- 172.21.15.26:0/3123705223 learned_addr learned my addr 172.21.15.26:0/3123705223 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:10.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.611+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270728676 0 0) 0x7ff4480b28a0 con 0x7ff4481430c0 2024-03-25T14:36:10.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.611+0000 7ff44ce25640 10 monclient(hunting): _negotiate 2024-03-25T14:36:10.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.611+0000 7ff44ce25640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:10.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.611+0000 7ff44ce25640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:10.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.611+0000 7ff44ce25640 10 monclient(hunting): my global_id is 4621 2024-03-25T14:36:10.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.611+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff434003520 con 0x7ff4481430c0 2024-03-25T14:36:10.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.613+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242192891 0 0) 0x7ff4480b2a30 con 0x7ff4480a98f0 2024-03-25T14:36:10.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.613+0000 7ff44ce25640 10 monclient(hunting): _negotiate 2024-03-25T14:36:10.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.613+0000 7ff44ce25640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:10.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.613+0000 7ff44ce25640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:10.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.613+0000 7ff44ce25640 10 monclient(hunting): my global_id is 4583 2024-03-25T14:36:10.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.613+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4480b28a0 con 0x7ff4480a98f0 2024-03-25T14:36:10.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 743911270 0 0) 0x7ff434003520 con 0x7ff4481430c0 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 >> v1:172.21.15.26:6790/0 conn(0x7ff4480a98f0 legacy=0x7ff4480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 >> v1:172.21.15.26:6789/0 conn(0x7ff44813f710 legacy=0x7ff4481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: _finish_hunting 0 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 1 monclient: found mon.noname-c 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4480b44c0 con 0x7ff4481430c0 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: _finish_auth 0 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: _check_auth_tickets 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff440001a10 con 0x7ff4481430c0 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:10.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff44ce25640 10 monclient: dump: 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:10.623 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:10.624 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:10.624 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:10.624 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:10.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff45008b640 5 monclient: authenticate success, global_id 4621 2024-03-25T14:36:10.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.616+0000 7ff45008b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:10.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff440001b70 con 0x7ff4481430c0 2024-03-25T14:36:10.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff44ce25640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:10.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff44ce25640 1 -- 172.21.15.26:0/3123705223 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff440001f80 con 0x7ff4481430c0 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff44ce25640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff44ce25640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff44ce25640 10 monclient: dump: 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:10.640 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:10.641 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:10.641 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:10.641 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:10.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff45008b640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:10.641 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.634+0000 7ff45008b640 4 set_mon_vals no callback set 2024-03-25T14:36:10.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 10 monclient: shutdown 2024-03-25T14:36:10.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:10.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 1 -- 172.21.15.26:0/3123705223 >> v1:172.21.15.121:6789/0 conn(0x7ff4481430c0 legacy=0x7ff44814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 1 -- 172.21.15.26:0/3123705223 shutdown_connections 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 1 -- 172.21.15.26:0/3123705223 >> 172.21.15.26:0/3123705223 conn(0x7ff4480b7ea0 msgr2=0x7ff4480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 1 -- 172.21.15.26:0/3123705223 shutdown_connections 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.635+0000 7ff45008b640 1 -- 172.21.15.26:0/3123705223 wait complete. 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.636+0000 7ff45008b640 10 monclient: build_initial_monmap 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.636+0000 7ff45008b640 10 monclient: monmap: 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:10.644 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 1 librados: starting msgr at 2024-03-25T14:36:10.645 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 1 librados: starting objecter 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 1 Processor -- start 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 1 -- start start 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 1 librados: setting wanted keys 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 1 librados: calling monclient init 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.637+0000 7ff45008b640 10 monclient: init 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:10.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:36:10.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4480a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:10.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient(hunting): picked mon.noname-c con 0x7ff44813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:10.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4481430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:10.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff448136ec0 con 0x7ff4480a98f0 2024-03-25T14:36:10.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff448136c20 con 0x7ff4481430c0 2024-03-25T14:36:10.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff448137db0 con 0x7ff44813f710 2024-03-25T14:36:10.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:10.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.638+0000 7ff45008b640 10 monclient(hunting): authenticate will time out at 2055.741876s 2024-03-25T14:36:10.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.641+0000 7ff44de27640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff4480a98f0 0x7ff4481376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:54386/0 (socket says 172.21.15.26:54386) 2024-03-25T14:36:10.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.641+0000 7ff44de27640 1 -- 172.21.15.26:0/2577854734 learned_addr learned my addr 172.21.15.26:0/2577854734 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:10.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.642+0000 7ff43e7fc640 10 client.?.objecter ms_handle_connect 0x7ff4480a98f0 2024-03-25T14:36:10.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.642+0000 7ff43e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:10.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.647+0000 7ff43e7fc640 10 client.?.objecter ms_handle_connect 0x7ff4481430c0 2024-03-25T14:36:10.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.647+0000 7ff43e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:10.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.650+0000 7ff43e7fc640 10 client.?.objecter ms_handle_connect 0x7ff44813f710 2024-03-25T14:36:10.655 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.650+0000 7ff43e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:10.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.657+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139700266 0 0) 0x7ff448136ec0 con 0x7ff4480a98f0 2024-03-25T14:36:10.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.657+0000 7ff43e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:36:10.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.657+0000 7ff43e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:10.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.657+0000 7ff43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:10.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.657+0000 7ff43e7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T14:36:10.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.657+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42c004eb0 con 0x7ff4480a98f0 2024-03-25T14:36:10.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.658+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911086538 0 0) 0x7ff448136c20 con 0x7ff4481430c0 2024-03-25T14:36:10.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.658+0000 7ff43e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:36:10.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.658+0000 7ff43e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:10.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.658+0000 7ff43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:10.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.658+0000 7ff43e7fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T14:36:10.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.658+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff448136ec0 con 0x7ff4481430c0 2024-03-25T14:36:10.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.667+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620585600 0 0) 0x7ff448137db0 con 0x7ff44813f710 2024-03-25T14:36:10.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.667+0000 7ff43e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:36:10.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.667+0000 7ff43e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:10.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.667+0000 7ff43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:10.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.667+0000 7ff43e7fc640 10 monclient(hunting): my global_id is 4586 2024-03-25T14:36:10.673 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.667+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff448136c20 con 0x7ff44813f710 2024-03-25T14:36:10.674 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3595866652 0 0) 0x7ff448136ec0 con 0x7ff4481430c0 2024-03-25T14:36:10.674 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff448137db0 con 0x7ff4481430c0 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff440001e80 con 0x7ff4481430c0 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 10 monclient(hunting): dump: 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:10.675 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:10.676 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:10.676 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:10.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.669+0000 7ff43e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:10.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1001074358 0 0) 0x7ff448137db0 con 0x7ff4481430c0 2024-03-25T14:36:10.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 >> v1:172.21.15.26:6790/0 conn(0x7ff44813f710 legacy=0x7ff4480b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:10.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 >> v1:172.21.15.26:6789/0 conn(0x7ff4480a98f0 legacy=0x7ff4481376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:10.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:36:10.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 1 monclient: found mon.b 2024-03-25T14:36:10.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:10.683 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff448138b50 con 0x7ff4481430c0 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff43e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 5 monclient: authenticate success, global_id 4624 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 10 monclient: _renew_subs 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:10.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff448139830 con 0x7ff4481430c0 2024-03-25T14:36:10.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:10.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 10 monclient: _renew_subs 2024-03-25T14:36:10.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:10.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481369b0 con 0x7ff4481430c0 2024-03-25T14:36:10.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.677+0000 7ff45008b640 1 librados: init done 2024-03-25T14:36:10.686 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.678+0000 7ff45008b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:10.686 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.678+0000 7ff45008b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:10.686 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.678+0000 7ff45008b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:10.686 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.678+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480627b0 con 0x7ff4481430c0 2024-03-25T14:36:10.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff440002040 con 0x7ff4481430c0 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff440003dc0 con 0x7ff4481430c0 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43e7fc640 10 monclient: dump: 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:10.952 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:10.953 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:10.953 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:10.953 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:10.953 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:10.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.946+0000 7ff43ffff640 4 set_mon_vals no callback set 2024-03-25T14:36:10.953 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.947+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ff440019d90 con 0x7ff4481430c0 2024-03-25T14:36:10.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.948+0000 7ff44de27640 10 monclient: get_auth_request con 0x7ff42c06de20 auth_method 0 2024-03-25T14:36:10.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.948+0000 7ff43e7fc640 10 client.4624.objecter ms_handle_connect 0x7ff42c06de20 2024-03-25T14:36:10.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.951+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7ff4400538d0 con 0x7ff4481430c0 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.951+0000 7ff43e7fc640 10 client.4624.objecter ms_dispatch 0x7ff448000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.951+0000 7ff43e7fc640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.951+0000 7ff43e7fc640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.952+0000 7ff43e7fc640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.952+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff44004f3e0 con 0x7ff4481430c0 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.952+0000 7ff43e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:10.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:10.952+0000 7ff43e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:11.187 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.182+0000 7ff45008b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:11.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.182+0000 7ff45008b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:11.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.182+0000 7ff45008b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:11.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.182+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff448069f40 con 0x7ff4481430c0 2024-03-25T14:36:11.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.231+0000 7ff43e7fc640 1 -- 172.21.15.26:0/2577854734 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7ff440051310 con 0x7ff4481430c0 2024-03-25T14:36:11.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.231+0000 7ff43e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:11.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.231+0000 7ff43e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:11.236 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:11.237 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:11.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 10 librados: watch_flush enter 2024-03-25T14:36:11.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 10 librados: watch_flush exit 2024-03-25T14:36:11.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T14:36:11.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 10 client.4624.objecter successfully canceled tick 2024-03-25T14:36:11.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 >> v1:172.21.15.121:6800/288813066 conn(0x7ff42c06de20 legacy=0x7ff42c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 10 monclient: shutdown 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.233+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 >> v1:172.21.15.121:6789/0 conn(0x7ff4481430c0 legacy=0x7ff4481e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.234+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 shutdown_connections 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.234+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 >> 172.21.15.26:0/2577854734 conn(0x7ff4480b7ea0 msgr2=0x7ff44813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.234+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 shutdown_connections 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.234+0000 7ff45008b640 1 -- 172.21.15.26:0/2577854734 wait complete. 2024-03-25T14:36:11.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:11.234+0000 7ff45008b640 1 librados: shutdown 2024-03-25T14:36:11.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:18.063 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:18.181 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.177+0000 7f4a68406640 10 monclient: get_monmap_and_config 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.177+0000 7f4a68406640 10 monclient: build_initial_monmap 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.177+0000 7f4a68406640 10 monclient: monmap: 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:18.178428+0000 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:18.178428+0000 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:18.182 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:18.183 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.178+0000 7f4a68406640 1 Processor -- start 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.178+0000 7f4a68406640 1 -- start start 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.178+0000 7f4a68406640 10 monclient: init 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient(hunting): picked mon.noname-b con 0x7f4a600a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient(hunting): picked mon.noname-a con 0x7f4a6013f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:18.184 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient(hunting): picked mon.noname-c con 0x7f4a601430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:18.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a60142520 con 0x7f4a6013f710 2024-03-25T14:36:18.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600b28a0 con 0x7f4a601430c0 2024-03-25T14:36:18.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600b2a30 con 0x7f4a600a98f0 2024-03-25T14:36:18.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:18.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.179+0000 7f4a68406640 10 monclient(hunting): authenticate will time out at 2063.282007s 2024-03-25T14:36:18.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.184+0000 7f4a661a2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4a600a98f0 0x7f4a600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:42906/0 (socket says 172.21.15.26:42906) 2024-03-25T14:36:18.188 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.184+0000 7f4a661a2640 1 -- 172.21.15.26:0/1714392728 learned_addr learned my addr 172.21.15.26:0/1714392728 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:18.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.193+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661154979 0 0) 0x7f4a600b2a30 con 0x7f4a600a98f0 2024-03-25T14:36:18.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.193+0000 7f4a651a0640 10 monclient(hunting): _negotiate 2024-03-25T14:36:18.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.193+0000 7f4a651a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:18.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.193+0000 7f4a651a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:18.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.193+0000 7f4a651a0640 10 monclient(hunting): my global_id is 4589 2024-03-25T14:36:18.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.193+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4c003520 con 0x7f4a600a98f0 2024-03-25T14:36:18.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.204+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338933665 0 0) 0x7f4a60142520 con 0x7f4a6013f710 2024-03-25T14:36:18.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.204+0000 7f4a651a0640 10 monclient(hunting): _negotiate 2024-03-25T14:36:18.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.204+0000 7f4a651a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:18.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.204+0000 7f4a651a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:18.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.204+0000 7f4a651a0640 10 monclient(hunting): my global_id is 4644 2024-03-25T14:36:18.208 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.204+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a600b2a30 con 0x7f4a6013f710 2024-03-25T14:36:18.212 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.208+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554515318 0 0) 0x7f4a4c003520 con 0x7f4a600a98f0 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 >> v1:172.21.15.121:6789/0 conn(0x7f4a601430c0 legacy=0x7f4a6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 >> v1:172.21.15.26:6789/0 conn(0x7f4a6013f710 legacy=0x7f4a601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: _finish_hunting 0 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 1 monclient: found mon.noname-b 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a600b44c0 con 0x7f4a600a98f0 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: _finish_auth 0 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: _check_auth_tickets 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4a5c001690 con 0x7f4a600a98f0 2024-03-25T14:36:18.213 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a651a0640 10 monclient: dump: 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a68406640 5 monclient: authenticate success, global_id 4589 2024-03-25T14:36:18.214 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.209+0000 7f4a68406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:18.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a5c0017f0 con 0x7f4a600a98f0 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a651a0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a651a0640 1 -- 172.21.15.26:0/1714392728 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4a5c001c00 con 0x7f4a600a98f0 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a651a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a651a0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a651a0640 10 monclient: dump: 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.220 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a68406640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:18.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.215+0000 7f4a68406640 4 set_mon_vals no callback set 2024-03-25T14:36:18.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 10 monclient: shutdown 2024-03-25T14:36:18.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:18.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 1 -- 172.21.15.26:0/1714392728 >> v1:172.21.15.26:6790/0 conn(0x7f4a600a98f0 legacy=0x7f4a600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 1 -- 172.21.15.26:0/1714392728 shutdown_connections 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 1 -- 172.21.15.26:0/1714392728 >> 172.21.15.26:0/1714392728 conn(0x7f4a600b7ea0 msgr2=0x7f4a600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 1 -- 172.21.15.26:0/1714392728 shutdown_connections 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.216+0000 7f4a68406640 1 -- 172.21.15.26:0/1714392728 wait complete. 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.217+0000 7f4a68406640 10 monclient: build_initial_monmap 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.217+0000 7f4a68406640 10 monclient: monmap: 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:18.224 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.217+0000 7f4a68406640 1 librados: starting msgr at 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.217+0000 7f4a68406640 1 librados: starting objecter 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 Processor -- start 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 -- start start 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 librados: setting wanted keys 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 librados: calling monclient init 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient: init 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:36:18.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient(hunting): picked mon.noname-b con 0x7f4a600a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient(hunting): picked mon.noname-c con 0x7f4a6013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient(hunting): picked mon.noname-a con 0x7f4a601430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a60135610 con 0x7f4a601430c0 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a60137230 con 0x7f4a600a98f0 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a60135cf0 con 0x7f4a6013f710 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:18.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.218+0000 7f4a68406640 10 monclient(hunting): authenticate will time out at 2063.321477s 2024-03-25T14:36:18.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.225+0000 7f4a669a3640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4a601430c0 0x7f4a601e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51494/0 (socket says 172.21.15.26:51494) 2024-03-25T14:36:18.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.225+0000 7f4a669a3640 1 -- 172.21.15.26:0/590949128 learned_addr learned my addr 172.21.15.26:0/590949128 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:18.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.227+0000 7f4a56ffd640 10 client.?.objecter ms_handle_connect 0x7f4a601430c0 2024-03-25T14:36:18.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.227+0000 7f4a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:18.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.232+0000 7f4a56ffd640 10 client.?.objecter ms_handle_connect 0x7f4a6013f710 2024-03-25T14:36:18.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.232+0000 7f4a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:18.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.234+0000 7f4a56ffd640 10 client.?.objecter ms_handle_connect 0x7f4a600a98f0 2024-03-25T14:36:18.238 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.234+0000 7f4a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:18.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.241+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99023470 0 0) 0x7f4a60135610 con 0x7f4a601430c0 2024-03-25T14:36:18.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.241+0000 7f4a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:18.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.241+0000 7f4a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:18.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.241+0000 7f4a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:18.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.241+0000 7f4a56ffd640 10 monclient(hunting): my global_id is 4647 2024-03-25T14:36:18.245 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.241+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a44004eb0 con 0x7f4a601430c0 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789544093 0 0) 0x7f4a60137230 con 0x7f4a600a98f0 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a60135610 con 0x7f4a600a98f0 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146248863 0 0) 0x7f4a60135cf0 con 0x7f4a6013f710 2024-03-25T14:36:18.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:18.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:18.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:18.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 10 monclient(hunting): my global_id is 4592 2024-03-25T14:36:18.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.253+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a60137230 con 0x7f4a6013f710 2024-03-25T14:36:18.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190770214 0 0) 0x7f4a44004eb0 con 0x7f4a601430c0 2024-03-25T14:36:18.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a60135cf0 con 0x7f4a601430c0 2024-03-25T14:36:18.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4a58001c30 con 0x7f4a601430c0 2024-03-25T14:36:18.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 10 monclient(hunting): dump: 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:18.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.255+0000 7f4a56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:18.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 90638909 0 0) 0x7f4a60135cf0 con 0x7f4a601430c0 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 >> v1:172.21.15.26:6790/0 conn(0x7f4a6013f710 legacy=0x7f4a601dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 >> v1:172.21.15.121:6789/0 conn(0x7f4a600a98f0 legacy=0x7f4a60130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 1 monclient: found mon.a 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:18.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60135e80 con 0x7f4a601430c0 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 10 monclient: _finish_auth 0 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 5 monclient: authenticate success, global_id 4647 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 10 monclient: _renew_subs 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a60131660 con 0x7f4a601430c0 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 10 monclient: _renew_subs 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a600b7960 con 0x7f4a601430c0 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.272+0000 7f4a68406640 1 librados: init done 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.273+0000 7f4a68406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:18.279 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.273+0000 7f4a68406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:18.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.273+0000 7f4a68406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:18.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.273+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a600627b0 con 0x7f4a601430c0 2024-03-25T14:36:18.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.527+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a58003970 con 0x7f4a601430c0 2024-03-25T14:36:18.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.527+0000 7f4a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:18.531 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.527+0000 7f4a6499f640 4 set_mon_vals no callback set 2024-03-25T14:36:18.590 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.586+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4a58003d80 con 0x7f4a601430c0 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.586+0000 7f4a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.586+0000 7f4a56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.586+0000 7f4a56ffd640 10 monclient: dump: 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:18.591 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:18.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.587+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4a58003f60 con 0x7f4a601430c0 2024-03-25T14:36:18.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.588+0000 7f4a661a2640 10 monclient: get_auth_request con 0x7f4a4406dda0 auth_method 0 2024-03-25T14:36:18.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.588+0000 7f4a56ffd640 10 client.4647.objecter ms_handle_connect 0x7f4a4406dda0 2024-03-25T14:36:18.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f4a58054b10 con 0x7f4a601430c0 2024-03-25T14:36:18.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 10 client.4647.objecter ms_dispatch 0x7f4a60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:18.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 3 client.4647.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:18.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 3 client.4647.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:18.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 20 client.4647.objecter dump_active .. 0 homeless 2024-03-25T14:36:18.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a58050620 con 0x7f4a601430c0 2024-03-25T14:36:18.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:18.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.593+0000 7f4a56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:18.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.822+0000 7f4a68406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:18.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.822+0000 7f4a68406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:18.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.822+0000 7f4a68406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:36:18.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.822+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a600b7c50 con 0x7f4a601430c0 2024-03-25T14:36:18.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.841+0000 7f4a56ffd640 1 -- 172.21.15.26:0/590949128 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f4a58052550 con 0x7f4a601430c0 2024-03-25T14:36:18.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.841+0000 7f4a56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:18.845 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.841+0000 7f4a56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:18.846 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:18.846 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:18.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 10 librados: watch_flush enter 2024-03-25T14:36:18.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 10 librados: watch_flush exit 2024-03-25T14:36:18.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 20 client.4647.objecter shutdown clearing up homeless session... 2024-03-25T14:36:18.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 10 client.4647.objecter successfully canceled tick 2024-03-25T14:36:18.848 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 >> v1:172.21.15.121:6800/288813066 conn(0x7f4a4406dda0 legacy=0x7f4a44070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 10 monclient: shutdown 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 >> v1:172.21.15.26:6789/0 conn(0x7f4a601430c0 legacy=0x7f4a601e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 shutdown_connections 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 >> 172.21.15.26:0/590949128 conn(0x7f4a600b7ea0 msgr2=0x7f4a60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 shutdown_connections 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 -- 172.21.15.26:0/590949128 wait complete. 2024-03-25T14:36:18.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:18.844+0000 7f4a68406640 1 librados: shutdown 2024-03-25T14:36:18.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:22.505 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:41.366 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.514 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:41.375 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.539 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:09:41.400 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.581 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:41.442 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.581 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:41.459 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.585 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:41.463 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.597 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:41.458 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.612 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:09:41.490 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:22.627 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:09:41.488 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:25.666 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.774+0000 7f5298d25640 10 monclient: get_monmap_and_config 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.774+0000 7f5298d25640 10 monclient: build_initial_monmap 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.774+0000 7f5298d25640 10 monclient: monmap: 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:25.774970+0000 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:25.774970+0000 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:25.778 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:25.779 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:25.779 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:25.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.774+0000 7f5298d25640 1 Processor -- start 2024-03-25T14:36:25.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.774+0000 7f5298d25640 1 -- start start 2024-03-25T14:36:25.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.774+0000 7f5298d25640 10 monclient: init 2024-03-25T14:36:25.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:25.780 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient(hunting): picked mon.noname-a con 0x7f52940b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient(hunting): picked mon.noname-c con 0x7f52940b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient(hunting): picked mon.noname-b con 0x7f529414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940af940 con 0x7f52940b4d60 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940b0a70 con 0x7f52940b1730 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940b0d50 con 0x7f529414be90 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.775+0000 7f5298d25640 10 monclient(hunting): authenticate will time out at 2070.878136s 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.776+0000 7f5291d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f52940b1730 0x7f52940b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:41134/0 (socket says 172.21.15.26:41134) 2024-03-25T14:36:25.781 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.776+0000 7f5291d9b640 1 -- 172.21.15.26:0/1874847644 learned_addr learned my addr 172.21.15.26:0/1874847644 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:25.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.847+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242745372 0 0) 0x7f52940af940 con 0x7f52940b4d60 2024-03-25T14:36:25.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.847+0000 7f529159a640 10 monclient(hunting): _negotiate 2024-03-25T14:36:25.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.847+0000 7f529159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:25.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.847+0000 7f529159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:25.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.847+0000 7f529159a640 10 monclient(hunting): my global_id is 4650 2024-03-25T14:36:25.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.847+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5280003520 con 0x7f52940b4d60 2024-03-25T14:36:25.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088123667 0 0) 0x7f52940b0a70 con 0x7f52940b1730 2024-03-25T14:36:25.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient(hunting): _negotiate 2024-03-25T14:36:25.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:25.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient(hunting): my global_id is 4633 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52940af940 con 0x7f52940b1730 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608342736 0 0) 0x7f5280003520 con 0x7f52940b4d60 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 >> v1:172.21.15.26:6790/0 conn(0x7f529414be90 legacy=0x7f529414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 >> v1:172.21.15.121:6789/0 conn(0x7f52940b1730 legacy=0x7f52940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: _finish_hunting 0 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 monclient: found mon.noname-a 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940b1030 con 0x7f52940b4d60 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: _finish_auth 0 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: _check_auth_tickets 2024-03-25T14:36:25.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5284001a10 con 0x7f52940b4d60 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f529159a640 10 monclient: dump: 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:25.870 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:25.871 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:25.871 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:25.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f5298d25640 5 monclient: authenticate success, global_id 4650 2024-03-25T14:36:25.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.863+0000 7f5298d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5284001b70 con 0x7f52940b4d60 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f529159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f529159a640 1 -- 172.21.15.26:0/1874847644 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5284001f80 con 0x7f52940b4d60 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f5298d25640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f5298d25640 4 set_mon_vals no callback set 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f529159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f529159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:36:25.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.880+0000 7f529159a640 10 monclient: dump: 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:25.885 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 10 monclient: shutdown 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 1 -- 172.21.15.26:0/1874847644 >> v1:172.21.15.26:6789/0 conn(0x7f52940b4d60 legacy=0x7f529413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 1 -- 172.21.15.26:0/1874847644 shutdown_connections 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 1 -- 172.21.15.26:0/1874847644 >> 172.21.15.26:0/1874847644 conn(0x7f52940617a0 msgr2=0x7f5294061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 1 -- 172.21.15.26:0/1874847644 shutdown_connections 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.881+0000 7f5298d25640 1 -- 172.21.15.26:0/1874847644 wait complete. 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 10 monclient: build_initial_monmap 2024-03-25T14:36:25.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 10 monclient: monmap: 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 1 librados: starting msgr at 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 1 librados: starting objecter 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 1 Processor -- start 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 1 -- start start 2024-03-25T14:36:25.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 1 librados: setting wanted keys 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 1 librados: calling monclient init 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.882+0000 7f5298d25640 10 monclient: init 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient(hunting): picked mon.noname-a con 0x7f52940b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient(hunting): picked mon.noname-c con 0x7f52940b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient(hunting): picked mon.noname-b con 0x7f529414be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5294136130 con 0x7f52940b1730 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5294145800 con 0x7f529414be90 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52941452b0 con 0x7f52940b4d60 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:25.890 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.883+0000 7f5298d25640 10 monclient(hunting): authenticate will time out at 2070.985812s 2024-03-25T14:36:25.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.891+0000 7f529259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f52940b1730 0x7f52941dac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:56006/0 (socket says 172.21.15.26:56006) 2024-03-25T14:36:25.895 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.891+0000 7f529259c640 1 -- 172.21.15.26:0/1813341619 learned_addr learned my addr 172.21.15.26:0/1813341619 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:25.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.893+0000 7f528affd640 10 client.?.objecter ms_handle_connect 0x7f52940b1730 2024-03-25T14:36:25.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.893+0000 7f528affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:25.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.893+0000 7f528affd640 10 client.?.objecter ms_handle_connect 0x7f529414be90 2024-03-25T14:36:25.897 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.893+0000 7f528affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:25.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.895+0000 7f528affd640 10 client.?.objecter ms_handle_connect 0x7f52940b4d60 2024-03-25T14:36:25.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.895+0000 7f528affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:25.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.900+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149375974 0 0) 0x7f5294136130 con 0x7f52940b1730 2024-03-25T14:36:25.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.901+0000 7f528affd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:25.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.901+0000 7f528affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:25.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.901+0000 7f528affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:25.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.901+0000 7f528affd640 10 monclient(hunting): my global_id is 4653 2024-03-25T14:36:25.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.901+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5274004eb0 con 0x7f52940b1730 2024-03-25T14:36:25.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.905+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302835548 0 0) 0x7f5294145800 con 0x7f529414be90 2024-03-25T14:36:25.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.905+0000 7f528affd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:25.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.905+0000 7f528affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:25.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.905+0000 7f528affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:25.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.905+0000 7f528affd640 10 monclient(hunting): my global_id is 4636 2024-03-25T14:36:25.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.905+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5294136130 con 0x7f529414be90 2024-03-25T14:36:25.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.906+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334142888 0 0) 0x7f52941452b0 con 0x7f52940b4d60 2024-03-25T14:36:25.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.906+0000 7f528affd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:25.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.906+0000 7f528affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:25.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.906+0000 7f528affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:25.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.906+0000 7f528affd640 10 monclient(hunting): my global_id is 4598 2024-03-25T14:36:25.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.906+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5294145800 con 0x7f52940b4d60 2024-03-25T14:36:25.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482074552 0 0) 0x7f5294136130 con 0x7f529414be90 2024-03-25T14:36:25.923 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52941452b0 con 0x7f529414be90 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f528c001780 con 0x7f529414be90 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 10 monclient(hunting): dump: 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:25.924 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:25.925 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:25.925 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.919+0000 7f528affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.927+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3195091834 0 0) 0x7f5274004eb0 con 0x7f52940b1730 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.927+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5294136130 con 0x7f52940b1730 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.927+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5284001a10 con 0x7f52940b1730 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.927+0000 7f528affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.927+0000 7f528affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.927+0000 7f528affd640 10 monclient(hunting): dump: 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:25.931 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:25.932 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:25.933 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:25.933 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:25.933 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3042390547 0 0) 0x7f52941452b0 con 0x7f529414be90 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 >> v1:172.21.15.26:6790/0 conn(0x7f52940b4d60 legacy=0x7f52941db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 >> v1:172.21.15.26:6789/0 conn(0x7f52940b1730 legacy=0x7f52941dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 10 monclient: _finish_hunting 0 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 1 monclient: found mon.b 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5294145440 con 0x7f529414be90 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 10 monclient: _finish_auth 0 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 10 monclient: _check_auth_tickets 2024-03-25T14:36:25.935 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f528affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.928+0000 7f5298d25640 5 monclient: authenticate success, global_id 4636 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 10 monclient: _renew_subs 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5294135c10 con 0x7f529414be90 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 10 monclient: _renew_subs 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f529413e3f0 con 0x7f529414be90 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.929+0000 7f5298d25640 1 librados: init done 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.930+0000 7f5298d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.930+0000 7f5298d25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.930+0000 7f5298d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:25.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:25.930+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52941362c0 con 0x7f529414be90 2024-03-25T14:36:26.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f528c001c30 con 0x7f529414be90 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f528affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f528c003790 con 0x7f529414be90 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f528affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f528affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f528affd640 10 monclient: dump: 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:26.219 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:26.220 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:26.220 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:26.220 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:26.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.214+0000 7f5290d99640 4 set_mon_vals no callback set 2024-03-25T14:36:26.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.215+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f528c0039b0 con 0x7f529414be90 2024-03-25T14:36:26.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.216+0000 7f529259c640 10 monclient: get_auth_request con 0x7f527406df70 auth_method 0 2024-03-25T14:36:26.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.216+0000 7f528affd640 10 client.4636.objecter ms_handle_connect 0x7f527406df70 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f528c053320 con 0x7f529414be90 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 10 client.4636.objecter ms_dispatch 0x7f5294000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f528c04ee30 con 0x7f529414be90 2024-03-25T14:36:26.224 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:26.225 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.220+0000 7f528affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:26.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.475+0000 7f5298d25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:26.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.475+0000 7f5298d25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:26.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.475+0000 7f5298d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:26.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.475+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52940627b0 con 0x7f529414be90 2024-03-25T14:36:26.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.501+0000 7f528affd640 1 -- 172.21.15.26:0/1813341619 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f528c050d60 con 0x7f529414be90 2024-03-25T14:36:26.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.501+0000 7f528affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:26.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.501+0000 7f528affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:26.505 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:26.506 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:26.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.503+0000 7f5298d25640 10 librados: watch_flush enter 2024-03-25T14:36:26.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 10 librados: watch_flush exit 2024-03-25T14:36:26.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T14:36:26.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 10 client.4636.objecter successfully canceled tick 2024-03-25T14:36:26.508 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 >> v1:172.21.15.121:6800/288813066 conn(0x7f527406df70 legacy=0x7f5274070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 10 monclient: shutdown 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 >> v1:172.21.15.121:6789/0 conn(0x7f529414be90 legacy=0x7f52941e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 shutdown_connections 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 >> 172.21.15.26:0/1813341619 conn(0x7f52940617a0 msgr2=0x7f52940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 shutdown_connections 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 -- 172.21.15.26:0/1813341619 wait complete. 2024-03-25T14:36:26.509 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:26.504+0000 7f5298d25640 1 librados: shutdown 2024-03-25T14:36:26.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:33.329 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.437+0000 7ff1a65c6640 10 monclient: get_monmap_and_config 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.437+0000 7ff1a65c6640 10 monclient: build_initial_monmap 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.437+0000 7ff1a65c6640 10 monclient: monmap: 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:33.438221+0000 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:33.438221+0000 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:33.442 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:33.443 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:33.443 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 1 Processor -- start 2024-03-25T14:36:33.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 1 -- start start 2024-03-25T14:36:33.443 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient: init 2024-03-25T14:36:33.444 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1a00a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff1a013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff1a01430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a0142520 con 0x7ff1a01430c0 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a00b28a0 con 0x7ff1a013f710 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a00b2a30 con 0x7ff1a00a98f0 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:33.445 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.438+0000 7ff1a65c6640 10 monclient(hunting): authenticate will time out at 2078.541424s 2024-03-25T14:36:33.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.442+0000 7ff19ffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff1a00a98f0 0x7ff1a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52262/0 (socket says 172.21.15.26:52262) 2024-03-25T14:36:33.446 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.442+0000 7ff19ffff640 1 -- 172.21.15.26:0/595879552 learned_addr learned my addr 172.21.15.26:0/595879552 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:33.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.450+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887251805 0 0) 0x7ff1a00b2a30 con 0x7ff1a00a98f0 2024-03-25T14:36:33.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.450+0000 7ff19effd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:33.454 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.450+0000 7ff19effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:33.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.450+0000 7ff19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:33.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.450+0000 7ff19effd640 10 monclient(hunting): my global_id is 4601 2024-03-25T14:36:33.455 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.450+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff188003520 con 0x7ff1a00a98f0 2024-03-25T14:36:33.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.456+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249130822 0 0) 0x7ff1a0142520 con 0x7ff1a01430c0 2024-03-25T14:36:33.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.456+0000 7ff19effd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:33.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.456+0000 7ff19effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:33.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.456+0000 7ff19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:33.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.456+0000 7ff19effd640 10 monclient(hunting): my global_id is 4656 2024-03-25T14:36:33.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.456+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a00b2a30 con 0x7ff1a01430c0 2024-03-25T14:36:33.461 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472089982 0 0) 0x7ff188003520 con 0x7ff1a00a98f0 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 >> v1:172.21.15.121:6789/0 conn(0x7ff1a013f710 legacy=0x7ff1a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 >> v1:172.21.15.26:6789/0 conn(0x7ff1a01430c0 legacy=0x7ff1a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: _finish_hunting 0 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 1 monclient: found mon.noname-b 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00b44c0 con 0x7ff1a00a98f0 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: _finish_auth 0 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: _check_auth_tickets 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:33.462 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff190001690 con 0x7ff1a00a98f0 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff1a65c6640 5 monclient: authenticate success, global_id 4601 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff1a65c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff19effd640 10 monclient: dump: 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:33.463 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:33.464 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.464 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.457+0000 7ff1a65c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:33.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1900017f0 con 0x7ff1a00a98f0 2024-03-25T14:36:33.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff19effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:33.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff19effd640 1 -- 172.21.15.26:0/595879552 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff190001c00 con 0x7ff1a00a98f0 2024-03-25T14:36:33.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff1a65c6640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:33.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff1a65c6640 4 set_mon_vals no callback set 2024-03-25T14:36:33.465 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff19effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff19effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.461+0000 7ff19effd640 10 monclient: dump: 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:33.466 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 10 monclient: shutdown 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 1 -- 172.21.15.26:0/595879552 >> v1:172.21.15.26:6790/0 conn(0x7ff1a00a98f0 legacy=0x7ff1a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 1 -- 172.21.15.26:0/595879552 shutdown_connections 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 1 -- 172.21.15.26:0/595879552 >> 172.21.15.26:0/595879552 conn(0x7ff1a00b7ea0 msgr2=0x7ff1a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 1 -- 172.21.15.26:0/595879552 shutdown_connections 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.462+0000 7ff1a65c6640 1 -- 172.21.15.26:0/595879552 wait complete. 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.463+0000 7ff1a65c6640 10 monclient: build_initial_monmap 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.463+0000 7ff1a65c6640 10 monclient: monmap: 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:33.468 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.463+0000 7ff1a65c6640 1 librados: starting msgr at 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.463+0000 7ff1a65c6640 1 librados: starting objecter 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.463+0000 7ff1a65c6640 1 Processor -- start 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 1 -- start start 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 1 librados: setting wanted keys 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 1 librados: calling monclient init 2024-03-25T14:36:33.469 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient: init 2024-03-25T14:36:33.470 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff1a00a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1a013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff1a01430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a013be70 con 0x7ff1a01430c0 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a0131660 con 0x7ff1a013f710 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a01317f0 con 0x7ff1a00a98f0 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:33.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.464+0000 7ff1a65c6640 10 monclient(hunting): authenticate will time out at 2078.567501s 2024-03-25T14:36:33.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.475+0000 7ff1a4b63640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff1a01430c0 0x7ff1a01e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53304/0 (socket says 172.21.15.26:53304) 2024-03-25T14:36:33.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.475+0000 7ff1a4b63640 1 -- 172.21.15.26:0/3053224962 learned_addr learned my addr 172.21.15.26:0/3053224962 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:33.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.475+0000 7ff19cff9640 10 client.?.objecter ms_handle_connect 0x7ff1a01430c0 2024-03-25T14:36:33.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.476+0000 7ff19cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:33.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.477+0000 7ff19cff9640 10 client.?.objecter ms_handle_connect 0x7ff1a013f710 2024-03-25T14:36:33.480 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.477+0000 7ff19cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:33.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.482+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409174003 0 0) 0x7ff1a013be70 con 0x7ff1a01430c0 2024-03-25T14:36:33.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.482+0000 7ff19cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:36:33.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.482+0000 7ff19cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:33.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.482+0000 7ff19cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:33.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.482+0000 7ff19cff9640 10 monclient(hunting): my global_id is 4659 2024-03-25T14:36:33.486 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.482+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff18c004eb0 con 0x7ff1a01430c0 2024-03-25T14:36:33.487 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.483+0000 7ff19cff9640 10 client.?.objecter ms_handle_connect 0x7ff1a00a98f0 2024-03-25T14:36:33.487 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.483+0000 7ff19cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:33.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.489+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115315885 0 0) 0x7ff1a0131660 con 0x7ff1a013f710 2024-03-25T14:36:33.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.489+0000 7ff19cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:36:33.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.489+0000 7ff19cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:33.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.489+0000 7ff19cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:33.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.489+0000 7ff19cff9640 10 monclient(hunting): my global_id is 4642 2024-03-25T14:36:33.493 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.489+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a013be70 con 0x7ff1a013f710 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.508+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200446651 0 0) 0x7ff1a01317f0 con 0x7ff1a00a98f0 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.508+0000 7ff19cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.508+0000 7ff19cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.508+0000 7ff19cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.508+0000 7ff19cff9640 10 monclient(hunting): my global_id is 4604 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.508+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a0131660 con 0x7ff1a00a98f0 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329759153 0 0) 0x7ff1a013be70 con 0x7ff1a013f710 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1a01317f0 con 0x7ff1a013f710 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff1940018a0 con 0x7ff1a013f710 2024-03-25T14:36:33.513 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 10 monclient(hunting): dump: 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.509+0000 7ff19cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:33.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.512+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782808718 0 0) 0x7ff18c004eb0 con 0x7ff1a01430c0 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.512+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1a013be70 con 0x7ff1a01430c0 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.512+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff198001d20 con 0x7ff1a01430c0 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.512+0000 7ff19cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.512+0000 7ff19cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.512+0000 7ff19cff9640 10 monclient(hunting): dump: 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:33.516 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:33.517 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3243756932 0 0) 0x7ff1a01317f0 con 0x7ff1a013f710 2024-03-25T14:36:33.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 >> v1:172.21.15.26:6790/0 conn(0x7ff1a00a98f0 legacy=0x7ff1a0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 >> v1:172.21.15.26:6789/0 conn(0x7ff1a01430c0 legacy=0x7ff1a01e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 10 monclient: _finish_hunting 0 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 1 monclient: found mon.b 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0131980 con 0x7ff1a013f710 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 10 monclient: _finish_auth 0 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 10 monclient: _check_auth_tickets 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff19cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 5 monclient: authenticate success, global_id 4642 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 10 monclient: _renew_subs 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:33.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a0135c60 con 0x7ff1a013f710 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 10 monclient: _renew_subs 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a00b7770 con 0x7ff1a013f710 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.516+0000 7ff1a65c6640 1 librados: init done 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.518+0000 7ff1a65c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.518+0000 7ff1a65c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:33.522 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.518+0000 7ff1a65c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:33.523 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.518+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a01371d0 con 0x7ff1a013f710 2024-03-25T14:36:33.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff194001d50 con 0x7ff1a013f710 2024-03-25T14:36:33.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:33.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ff1940037d0 con 0x7ff1a013f710 2024-03-25T14:36:33.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:33.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:33.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19cff9640 10 monclient: dump: 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:33.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.534+0000 7ff19e7fc640 4 set_mon_vals no callback set 2024-03-25T14:36:33.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.535+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ff194003970 con 0x7ff1a013f710 2024-03-25T14:36:33.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.535+0000 7ff19ffff640 10 monclient: get_auth_request con 0x7ff18c06df60 auth_method 0 2024-03-25T14:36:33.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.536+0000 7ff19cff9640 10 client.4642.objecter ms_handle_connect 0x7ff18c06df60 2024-03-25T14:36:33.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.538+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7ff194053600 con 0x7ff1a013f710 2024-03-25T14:36:33.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.538+0000 7ff19cff9640 10 client.4642.objecter ms_dispatch 0x7ff1a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:33.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.538+0000 7ff19cff9640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:33.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.538+0000 7ff19cff9640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:33.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.538+0000 7ff19cff9640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T14:36:33.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.808+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff19404ef40 con 0x7ff1a013f710 2024-03-25T14:36:33.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.808+0000 7ff19cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:33.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:33.808+0000 7ff19cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:34.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.059+0000 7ff1a65c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:34.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.059+0000 7ff1a65c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:34.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.059+0000 7ff1a65c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:34.063 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.059+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1a0074cc0 con 0x7ff1a013f710 2024-03-25T14:36:34.093 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.089+0000 7ff19cff9640 1 -- 172.21.15.26:0/3053224962 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7ff194051040 con 0x7ff1a013f710 2024-03-25T14:36:34.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.090+0000 7ff19cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:34.094 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.090+0000 7ff19cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:34.094 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:34.094 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 10 librados: watch_flush enter 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 10 librados: watch_flush exit 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 10 client.4642.objecter successfully canceled tick 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 >> v1:172.21.15.121:6800/288813066 conn(0x7ff18c06df60 legacy=0x7ff18c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 10 monclient: shutdown 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.092+0000 7ff1a65c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:34.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.093+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 >> v1:172.21.15.121:6789/0 conn(0x7ff1a013f710 legacy=0x7ff1a01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:34.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.093+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 shutdown_connections 2024-03-25T14:36:34.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.093+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 >> 172.21.15.26:0/3053224962 conn(0x7ff1a00b7ea0 msgr2=0x7ff1a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:34.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.093+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 shutdown_connections 2024-03-25T14:36:34.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.093+0000 7ff1a65c6640 1 -- 172.21.15.26:0/3053224962 wait complete. 2024-03-25T14:36:34.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:34.093+0000 7ff1a65c6640 1 librados: shutdown 2024-03-25T14:36:34.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:40.918 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.022+0000 7fc92a9f4640 10 monclient: get_monmap_and_config 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.022+0000 7fc92a9f4640 10 monclient: build_initial_monmap 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.022+0000 7fc92a9f4640 10 monclient: monmap: 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:41.022927+0000 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:41.022927+0000 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:41.026 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:41.027 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:41.027 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:41.027 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:41.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.022+0000 7fc92a9f4640 1 Processor -- start 2024-03-25T14:36:41.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.022+0000 7fc92a9f4640 1 -- start start 2024-03-25T14:36:41.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.022+0000 7fc92a9f4640 10 monclient: init 2024-03-25T14:36:41.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:41.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:36:41.028 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient(hunting): picked mon.noname-a con 0x7fc9240a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient(hunting): picked mon.noname-b con 0x7fc92413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient(hunting): picked mon.noname-c con 0x7fc9241430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc924142520 con 0x7fc9240a98f0 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9240b28a0 con 0x7fc9241430c0 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9240b2a30 con 0x7fc92413f710 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:41.029 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.023+0000 7fc92a9f4640 10 monclient(hunting): authenticate will time out at 2086.126346s 2024-03-25T14:36:41.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.029+0000 7fc928f91640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc9241430c0 0x7fc92414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42050/0 (socket says 172.21.15.26:42050) 2024-03-25T14:36:41.033 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.029+0000 7fc928f91640 1 -- 172.21.15.26:0/3602293041 learned_addr learned my addr 172.21.15.26:0/3602293041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:41.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.038+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151428165 0 0) 0x7fc9240b28a0 con 0x7fc9241430c0 2024-03-25T14:36:41.042 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.038+0000 7fc922ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:41.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.038+0000 7fc922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:41.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.038+0000 7fc922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:41.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.038+0000 7fc922ffd640 10 monclient(hunting): my global_id is 4645 2024-03-25T14:36:41.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.038+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90c003520 con 0x7fc9241430c0 2024-03-25T14:36:41.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.046+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449931703 0 0) 0x7fc90c003520 con 0x7fc9241430c0 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 >> v1:172.21.15.26:6790/0 conn(0x7fc92413f710 legacy=0x7fc9241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 >> v1:172.21.15.26:6789/0 conn(0x7fc9240a98f0 legacy=0x7fc9240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 1 monclient: found mon.noname-c 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240b44c0 con 0x7fc9241430c0 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: _finish_auth 0 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc91c001a10 con 0x7fc9241430c0 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:41.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc922ffd640 10 monclient: dump: 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc92a9f4640 5 monclient: authenticate success, global_id 4645 2024-03-25T14:36:41.052 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.047+0000 7fc92a9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc91c001be0 con 0x7fc9241430c0 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc922ffd640 1 -- 172.21.15.26:0/3602293041 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc91c002030 con 0x7fc9241430c0 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 4 set_mon_vals no callback set 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc922ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc922ffd640 10 monclient: dump: 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:41.055 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:41.056 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 10 monclient: shutdown 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 1 -- 172.21.15.26:0/3602293041 >> v1:172.21.15.121:6789/0 conn(0x7fc9241430c0 legacy=0x7fc92414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 1 -- 172.21.15.26:0/3602293041 shutdown_connections 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.051+0000 7fc92a9f4640 1 -- 172.21.15.26:0/3602293041 >> 172.21.15.26:0/3602293041 conn(0x7fc9240b7ea0 msgr2=0x7fc9240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.052+0000 7fc92a9f4640 1 -- 172.21.15.26:0/3602293041 shutdown_connections 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.052+0000 7fc92a9f4640 1 -- 172.21.15.26:0/3602293041 wait complete. 2024-03-25T14:36:41.058 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.052+0000 7fc92a9f4640 10 monclient: build_initial_monmap 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.052+0000 7fc92a9f4640 10 monclient: monmap: 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 1 librados: starting msgr at 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 1 librados: starting objecter 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 1 Processor -- start 2024-03-25T14:36:41.059 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 1 -- start start 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 1 librados: setting wanted keys 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 1 librados: calling monclient init 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.053+0000 7fc92a9f4640 10 monclient: init 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient(hunting): picked mon.noname-b con 0x7fc9240a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient(hunting): picked mon.noname-a con 0x7fc92413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient(hunting): picked mon.noname-c con 0x7fc9241430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc924138100 con 0x7fc92413f710 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9240abcc0 con 0x7fc9240a98f0 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9241d8320 con 0x7fc9241430c0 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:41.060 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.054+0000 7fc92a9f4640 10 monclient(hunting): authenticate will time out at 2086.156947s 2024-03-25T14:36:41.069 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.066+0000 7fc923fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc9240a98f0 0x7fc9241386b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42104/0 (socket says 172.21.15.26:42104) 2024-03-25T14:36:41.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.066+0000 7fc923fff640 1 -- 172.21.15.26:0/804276433 learned_addr learned my addr 172.21.15.26:0/804276433 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:41.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.067+0000 7fc920ff9640 10 client.?.objecter ms_handle_connect 0x7fc9240a98f0 2024-03-25T14:36:41.070 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.067+0000 7fc920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:41.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.072+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924601429 0 0) 0x7fc9240abcc0 con 0x7fc9240a98f0 2024-03-25T14:36:41.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.072+0000 7fc920ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:36:41.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.072+0000 7fc920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:41.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.072+0000 7fc920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:41.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.072+0000 7fc920ff9640 10 monclient(hunting): my global_id is 4648 2024-03-25T14:36:41.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.072+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc910004eb0 con 0x7fc9240a98f0 2024-03-25T14:36:41.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.073+0000 7fc920ff9640 10 client.?.objecter ms_handle_connect 0x7fc92413f710 2024-03-25T14:36:41.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.073+0000 7fc920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:41.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.078+0000 7fc920ff9640 10 client.?.objecter ms_handle_connect 0x7fc9241430c0 2024-03-25T14:36:41.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.078+0000 7fc920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:41.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.092+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191083527 0 0) 0x7fc9241d8320 con 0x7fc9241430c0 2024-03-25T14:36:41.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.092+0000 7fc920ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:36:41.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.092+0000 7fc920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:41.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.092+0000 7fc920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:41.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.092+0000 7fc920ff9640 10 monclient(hunting): my global_id is 4610 2024-03-25T14:36:41.097 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.092+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9240abcc0 con 0x7fc9241430c0 2024-03-25T14:36:41.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264206129 0 0) 0x7fc910004eb0 con 0x7fc9240a98f0 2024-03-25T14:36:41.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9241d8320 con 0x7fc9240a98f0 2024-03-25T14:36:41.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc914001960 con 0x7fc9240a98f0 2024-03-25T14:36:41.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:41.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:41.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 10 monclient(hunting): dump: 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:41.100 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.095+0000 7fc920ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:41.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.096+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102115080 0 0) 0x7fc924138100 con 0x7fc92413f710 2024-03-25T14:36:41.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.096+0000 7fc920ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:36:41.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.096+0000 7fc920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:41.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.096+0000 7fc920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:41.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.096+0000 7fc920ff9640 10 monclient(hunting): my global_id is 4662 2024-03-25T14:36:41.101 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.096+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc910004eb0 con 0x7fc92413f710 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 922467741 0 0) 0x7fc9241d8320 con 0x7fc9240a98f0 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 >> v1:172.21.15.26:6790/0 conn(0x7fc9241430c0 legacy=0x7fc9241e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 >> v1:172.21.15.26:6789/0 conn(0x7fc92413f710 legacy=0x7fc9241d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 1 monclient: found mon.b 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:41.119 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9241350d0 con 0x7fc9240a98f0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 10 monclient: _finish_auth 0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc92a9f4640 5 monclient: authenticate success, global_id 4648 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc92a9f4640 10 monclient: _renew_subs 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc92a9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.114+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9241353a0 con 0x7fc9240a98f0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.115+0000 7fc92a9f4640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.115+0000 7fc92a9f4640 10 monclient: _renew_subs 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.115+0000 7fc92a9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.115+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc924137b60 con 0x7fc9240a98f0 2024-03-25T14:36:41.120 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.115+0000 7fc92a9f4640 1 librados: init done 2024-03-25T14:36:41.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.116+0000 7fc92a9f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:41.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.116+0000 7fc92a9f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:41.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.116+0000 7fc92a9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:41.121 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.116+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc924137e60 con 0x7fc9240a98f0 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9140035f0 con 0x7fc9240a98f0 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc920ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fc914003940 con 0x7fc9240a98f0 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc920ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc920ff9640 10 monclient: dump: 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:41.405 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:41.406 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.401+0000 7fc9227fc640 4 set_mon_vals no callback set 2024-03-25T14:36:41.407 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.402+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fc914003b60 con 0x7fc9240a98f0 2024-03-25T14:36:41.407 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.402+0000 7fc9237fe640 10 monclient: get_auth_request con 0x7fc91006de20 auth_method 0 2024-03-25T14:36:41.407 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.403+0000 7fc920ff9640 10 client.4648.objecter ms_handle_connect 0x7fc91006de20 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fc9140547b0 con 0x7fc9240a98f0 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 10 client.4648.objecter ms_dispatch 0x7fc924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9140502c0 con 0x7fc9240a98f0 2024-03-25T14:36:41.410 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:41.411 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.406+0000 7fc920ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:41.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.660+0000 7fc92a9f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:41.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.660+0000 7fc92a9f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:41.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.660+0000 7fc92a9f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:41.665 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.660+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9240627b0 con 0x7fc9240a98f0 2024-03-25T14:36:41.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.690+0000 7fc920ff9640 1 -- 172.21.15.26:0/804276433 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fc9140521f0 con 0x7fc9240a98f0 2024-03-25T14:36:41.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.690+0000 7fc920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:41.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.690+0000 7fc920ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:41.694 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:41.694 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:41.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 10 librados: watch_flush enter 2024-03-25T14:36:41.696 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 10 librados: watch_flush exit 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 10 client.4648.objecter successfully canceled tick 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 >> v1:172.21.15.121:6800/288813066 conn(0x7fc91006de20 legacy=0x7fc9100702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 10 monclient: shutdown 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 >> v1:172.21.15.121:6789/0 conn(0x7fc9240a98f0 legacy=0x7fc9241386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 shutdown_connections 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.692+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 >> 172.21.15.26:0/804276433 conn(0x7fc9240b7ea0 msgr2=0x7fc924074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.693+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 shutdown_connections 2024-03-25T14:36:41.697 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.693+0000 7fc92a9f4640 1 -- 172.21.15.26:0/804276433 wait complete. 2024-03-25T14:36:41.698 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:41.693+0000 7fc92a9f4640 1 librados: shutdown 2024-03-25T14:36:41.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:42.575 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:01.436 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.589 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:01.451 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.626 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:01.487 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.664 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:01.526 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.669 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:01.531 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.669 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:01.547 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.676 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:01.553 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.698 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:01.573 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:42.716 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:01.577 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:36:48.516 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:48.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.629+0000 7fd8c0da8640 10 monclient: get_monmap_and_config 2024-03-25T14:36:48.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.629+0000 7fd8c0da8640 10 monclient: build_initial_monmap 2024-03-25T14:36:48.633 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.629+0000 7fd8c0da8640 10 monclient: monmap: 2024-03-25T14:36:48.633 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:48.633 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:48.633 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:48.630533+0000 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:48.630533+0000 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:48.634 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:48.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.630+0000 7fd8c0da8640 1 Processor -- start 2024-03-25T14:36:48.635 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.630+0000 7fd8c0da8640 1 -- start start 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.630+0000 7fd8c0da8640 10 monclient: init 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8bc0a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8bc13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient(hunting): picked mon.noname-a con 0x7fd8bc1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc142520 con 0x7fd8bc1430c0 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0b28a0 con 0x7fd8bc13f710 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0b2a30 con 0x7fd8bc0a98f0 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:48.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.631+0000 7fd8c0da8640 10 monclient(hunting): authenticate will time out at 2093.733922s 2024-03-25T14:36:48.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.636+0000 7fd8b9d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd8bc13f710 0x7fd8bc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:33924/0 (socket says 172.21.15.26:33924) 2024-03-25T14:36:48.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.636+0000 7fd8b9d9b640 1 -- 172.21.15.26:0/3692144023 learned_addr learned my addr 172.21.15.26:0/3692144023 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:48.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.642+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775640853 0 0) 0x7fd8bc0b28a0 con 0x7fd8bc13f710 2024-03-25T14:36:48.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.642+0000 7fd8b959a640 10 monclient(hunting): _negotiate 2024-03-25T14:36:48.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.642+0000 7fd8b959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:48.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.642+0000 7fd8b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:48.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.642+0000 7fd8b959a640 10 monclient(hunting): my global_id is 4651 2024-03-25T14:36:48.646 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.642+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4003520 con 0x7fd8bc13f710 2024-03-25T14:36:48.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.655+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632725373 0 0) 0x7fd8bc142520 con 0x7fd8bc1430c0 2024-03-25T14:36:48.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.655+0000 7fd8b959a640 10 monclient(hunting): _negotiate 2024-03-25T14:36:48.658 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.655+0000 7fd8b959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:48.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.655+0000 7fd8b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:48.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.655+0000 7fd8b959a640 10 monclient(hunting): my global_id is 4665 2024-03-25T14:36:48.659 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.655+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc0b28a0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2096911274 0 0) 0x7fd8a4003520 con 0x7fd8bc13f710 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 >> v1:172.21.15.26:6790/0 conn(0x7fd8bc0a98f0 legacy=0x7fd8bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 >> v1:172.21.15.26:6789/0 conn(0x7fd8bc1430c0 legacy=0x7fd8bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: _finish_hunting 0 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 1 monclient: found mon.noname-c 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0b44c0 con 0x7fd8bc13f710 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: _finish_auth 0 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: _check_auth_tickets 2024-03-25T14:36:48.662 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fd8b0001690 con 0x7fd8bc13f710 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8c0da8640 5 monclient: authenticate success, global_id 4651 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8c0da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8b959a640 10 monclient: dump: 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:48.663 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:48.664 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:48.664 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:48.664 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:48.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.658+0000 7fd8c0da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:48.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b00017f0 con 0x7fd8bc13f710 2024-03-25T14:36:48.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:48.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8b959a640 1 -- 172.21.15.26:0/3692144023 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fd8b0001c00 con 0x7fd8bc13f710 2024-03-25T14:36:48.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8c0da8640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:48.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8c0da8640 4 set_mon_vals no callback set 2024-03-25T14:36:48.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.671+0000 7fd8b959a640 10 monclient: dump: 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:48.676 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:48.679 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 10 monclient: shutdown 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 1 -- 172.21.15.26:0/3692144023 >> v1:172.21.15.121:6789/0 conn(0x7fd8bc13f710 legacy=0x7fd8bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 1 -- 172.21.15.26:0/3692144023 shutdown_connections 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 1 -- 172.21.15.26:0/3692144023 >> 172.21.15.26:0/3692144023 conn(0x7fd8bc0b7ea0 msgr2=0x7fd8bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 1 -- 172.21.15.26:0/3692144023 shutdown_connections 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.672+0000 7fd8c0da8640 1 -- 172.21.15.26:0/3692144023 wait complete. 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.673+0000 7fd8c0da8640 10 monclient: build_initial_monmap 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.673+0000 7fd8c0da8640 10 monclient: monmap: 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:48.680 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 1 librados: starting msgr at 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 1 librados: starting objecter 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 1 Processor -- start 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 1 -- start start 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 1 librados: setting wanted keys 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 1 librados: calling monclient init 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.674+0000 7fd8c0da8640 10 monclient: init 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:48.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8bc0a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient(hunting): picked mon.noname-a con 0x7fd8bc13f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8bc1430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc138a50 con 0x7fd8bc13f710 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0ab9e0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc1d7880 con 0x7fd8bc0a98f0 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:48.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.675+0000 7fd8c0da8640 10 monclient(hunting): authenticate will time out at 2093.778278s 2024-03-25T14:36:48.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.684+0000 7fd8bad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd8bc1430c0 0x7fd8bc1e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:33972/0 (socket says 172.21.15.26:33972) 2024-03-25T14:36:48.688 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.684+0000 7fd8bad9d640 1 -- 172.21.15.26:0/2564711604 learned_addr learned my addr 172.21.15.26:0/2564711604 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:48.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.686+0000 7fd8aaffd640 10 client.?.objecter ms_handle_connect 0x7fd8bc0a98f0 2024-03-25T14:36:48.690 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.686+0000 7fd8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:48.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.687+0000 7fd8aaffd640 10 client.?.objecter ms_handle_connect 0x7fd8bc1430c0 2024-03-25T14:36:48.691 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.687+0000 7fd8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:48.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.689+0000 7fd8aaffd640 10 client.?.objecter ms_handle_connect 0x7fd8bc13f710 2024-03-25T14:36:48.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.689+0000 7fd8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:48.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.691+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561989313 0 0) 0x7fd8bc0ab9e0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.691+0000 7fd8aaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:48.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.691+0000 7fd8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:48.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.691+0000 7fd8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:48.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.691+0000 7fd8aaffd640 10 monclient(hunting): my global_id is 4654 2024-03-25T14:36:48.695 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.691+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89c004eb0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.697+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014862006 0 0) 0x7fd8bc1d7880 con 0x7fd8bc0a98f0 2024-03-25T14:36:48.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.697+0000 7fd8aaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:48.700 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.697+0000 7fd8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:48.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.697+0000 7fd8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:48.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.697+0000 7fd8aaffd640 10 monclient(hunting): my global_id is 4616 2024-03-25T14:36:48.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.697+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc0ab9e0 con 0x7fd8bc0a98f0 2024-03-25T14:36:48.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.702+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017127976 0 0) 0x7fd8bc138a50 con 0x7fd8bc13f710 2024-03-25T14:36:48.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.702+0000 7fd8aaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:36:48.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.702+0000 7fd8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:48.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.702+0000 7fd8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:48.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.702+0000 7fd8aaffd640 10 monclient(hunting): my global_id is 4668 2024-03-25T14:36:48.706 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.702+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc1d7880 con 0x7fd8bc13f710 2024-03-25T14:36:48.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908163731 0 0) 0x7fd89c004eb0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8bc138a50 con 0x7fd8bc1430c0 2024-03-25T14:36:48.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fd8b4001c70 con 0x7fd8bc1430c0 2024-03-25T14:36:48.709 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 10 monclient(hunting): dump: 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:48.710 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.705+0000 7fd8aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:48.716 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746222453 0 0) 0x7fd8bc138a50 con 0x7fd8bc1430c0 2024-03-25T14:36:48.716 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 >> v1:172.21.15.26:6790/0 conn(0x7fd8bc0a98f0 legacy=0x7fd8bc138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:48.716 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 >> v1:172.21.15.26:6789/0 conn(0x7fd8bc13f710 legacy=0x7fd8bc1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 10 monclient: _finish_hunting 0 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 1 monclient: found mon.b 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc1d84b0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 10 monclient: _finish_auth 0 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 10 monclient: _check_auth_tickets 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8c0da8640 5 monclient: authenticate success, global_id 4654 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8c0da8640 10 monclient: _renew_subs 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8c0da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.712+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc1d51f0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.713+0000 7fd8c0da8640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:48.717 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.713+0000 7fd8c0da8640 10 monclient: _renew_subs 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.713+0000 7fd8c0da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.713+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc138e60 con 0x7fd8bc1430c0 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.713+0000 7fd8c0da8640 1 librados: init done 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.714+0000 7fd8c0da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.714+0000 7fd8c0da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.714+0000 7fd8c0da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:48.718 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.714+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc1d76e0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.993 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b4003970 con 0x7fd8bc1430c0 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fd8b4003ba0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8aaffd640 10 monclient: dump: 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:48.994 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:48.995 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:48.995 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.989+0000 7fd8b8d99640 4 set_mon_vals no callback set 2024-03-25T14:36:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.991+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fd8b4003dc0 con 0x7fd8bc1430c0 2024-03-25T14:36:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.991+0000 7fd8ba59c640 10 monclient: get_auth_request con 0x7fd89c06ddf0 auth_method 0 2024-03-25T14:36:48.995 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.992+0000 7fd8aaffd640 10 client.4654.objecter ms_handle_connect 0x7fd89c06ddf0 2024-03-25T14:36:49.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.997+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7fd8b4054970 con 0x7fd8bc1430c0 2024-03-25T14:36:49.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.997+0000 7fd8aaffd640 10 client.4654.objecter ms_dispatch 0x7fd8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:49.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.997+0000 7fd8aaffd640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:49.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.997+0000 7fd8aaffd640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:49.001 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.997+0000 7fd8aaffd640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T14:36:49.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.999+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8b4050480 con 0x7fd8bc1430c0 2024-03-25T14:36:49.002 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.999+0000 7fd8aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:49.003 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:48.999+0000 7fd8aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:49.234 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.228+0000 7fd8c0da8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:49.234 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.228+0000 7fd8c0da8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:49.234 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.228+0000 7fd8c0da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:36:49.234 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.228+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8bc0627b0 con 0x7fd8bc1430c0 2024-03-25T14:36:49.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.249+0000 7fd8aaffd640 1 -- 172.21.15.26:0/2564711604 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7fd8b40523b0 con 0x7fd8bc1430c0 2024-03-25T14:36:49.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.249+0000 7fd8aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:49.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.249+0000 7fd8aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:49.253 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:49.253 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.251+0000 7fd8c0da8640 10 librados: watch_flush enter 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.251+0000 7fd8c0da8640 10 librados: watch_flush exit 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.251+0000 7fd8c0da8640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.251+0000 7fd8c0da8640 10 client.4654.objecter successfully canceled tick 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 >> v1:172.21.15.121:6800/288813066 conn(0x7fd89c06ddf0 legacy=0x7fd89c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 10 monclient: shutdown 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 >> v1:172.21.15.121:6789/0 conn(0x7fd8bc1430c0 legacy=0x7fd8bc1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 shutdown_connections 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 >> 172.21.15.26:0/2564711604 conn(0x7fd8bc0b7ea0 msgr2=0x7fd8bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 shutdown_connections 2024-03-25T14:36:49.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 -- 172.21.15.26:0/2564711604 wait complete. 2024-03-25T14:36:49.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:49.252+0000 7fd8c0da8640 1 librados: shutdown 2024-03-25T14:36:49.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:36:56.075 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:36:56.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.193+0000 7f393bd9d640 10 monclient: get_monmap_and_config 2024-03-25T14:36:56.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.193+0000 7f393bd9d640 10 monclient: build_initial_monmap 2024-03-25T14:36:56.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.193+0000 7f393bd9d640 10 monclient: monmap: 2024-03-25T14:36:56.197 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:56.197 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:56.197 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:36:56.194490+0000 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:36:56.194490+0000 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:36:56.198 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.194+0000 7f393bd9d640 1 Processor -- start 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.194+0000 7f393bd9d640 1 -- start start 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.194+0000 7f393bd9d640 10 monclient: init 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f39340a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:56.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f393413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:56.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f39341430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:56.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934142520 con 0x7f393413f710 2024-03-25T14:36:56.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340b28a0 con 0x7f39341430c0 2024-03-25T14:36:56.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340b2a30 con 0x7f39340a98f0 2024-03-25T14:36:56.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:56.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.195+0000 7f393bd9d640 10 monclient(hunting): authenticate will time out at 2101.297804s 2024-03-25T14:36:56.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.205+0000 7f393a33a640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f39341430c0 0x7f393414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:38598/0 (socket says 172.21.15.26:38598) 2024-03-25T14:36:56.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.205+0000 7f393a33a640 1 -- 172.21.15.26:0/2780427922 learned_addr learned my addr 172.21.15.26:0/2780427922 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:56.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.213+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225785400 0 0) 0x7f39340b28a0 con 0x7f39341430c0 2024-03-25T14:36:56.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.213+0000 7f3938b37640 10 monclient(hunting): _negotiate 2024-03-25T14:36:56.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.213+0000 7f3938b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:56.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.213+0000 7f3938b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:56.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.213+0000 7f3938b37640 10 monclient(hunting): my global_id is 4657 2024-03-25T14:36:56.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.213+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3920003520 con 0x7f39341430c0 2024-03-25T14:36:56.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.223+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811544832 0 0) 0x7f39340b2a30 con 0x7f39340a98f0 2024-03-25T14:36:56.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.223+0000 7f3938b37640 10 monclient(hunting): _negotiate 2024-03-25T14:36:56.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.223+0000 7f3938b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:56.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.223+0000 7f3938b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:56.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.223+0000 7f3938b37640 10 monclient(hunting): my global_id is 4619 2024-03-25T14:36:56.227 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.223+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39340b28a0 con 0x7f39340a98f0 2024-03-25T14:36:56.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.237+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345265381 0 0) 0x7f3934142520 con 0x7f393413f710 2024-03-25T14:36:56.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.237+0000 7f3938b37640 10 monclient(hunting): _negotiate 2024-03-25T14:36:56.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.237+0000 7f3938b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:56.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.237+0000 7f3938b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:56.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.237+0000 7f3938b37640 10 monclient(hunting): my global_id is 4671 2024-03-25T14:36:56.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.237+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39340b2a30 con 0x7f393413f710 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608823692 0 0) 0x7f39340b28a0 con 0x7f39340a98f0 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 >> v1:172.21.15.121:6789/0 conn(0x7f39341430c0 legacy=0x7f393414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 >> v1:172.21.15.26:6789/0 conn(0x7f393413f710 legacy=0x7f39341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: _finish_hunting 0 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 1 monclient: found mon.noname-b 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340b44c0 con 0x7f39340a98f0 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: _finish_auth 0 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: _check_auth_tickets 2024-03-25T14:36:56.248 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3930001670 con 0x7f39340a98f0 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f3938b37640 10 monclient: dump: 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:56.249 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:56.250 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:56.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f393bd9d640 5 monclient: authenticate success, global_id 4619 2024-03-25T14:36:56.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.242+0000 7f393bd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:36:56.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f3938b37640 1 -- 172.21.15.26:0/2780427922 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39300017d0 con 0x7f39340a98f0 2024-03-25T14:36:56.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f3938b37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:56.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 10 monclient: get_monmap_and_config success 2024-03-25T14:36:56.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 4 set_mon_vals no callback set 2024-03-25T14:36:56.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 10 monclient: shutdown 2024-03-25T14:36:56.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:56.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 1 -- 172.21.15.26:0/2780427922 >> v1:172.21.15.26:6790/0 conn(0x7f39340a98f0 legacy=0x7f39340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 1 -- 172.21.15.26:0/2780427922 shutdown_connections 2024-03-25T14:36:56.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.260+0000 7f393bd9d640 1 -- 172.21.15.26:0/2780427922 >> 172.21.15.26:0/2780427922 conn(0x7f39340b7ea0 msgr2=0x7f39340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:56.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.261+0000 7f393bd9d640 1 -- 172.21.15.26:0/2780427922 shutdown_connections 2024-03-25T14:36:56.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.261+0000 7f393bd9d640 1 -- 172.21.15.26:0/2780427922 wait complete. 2024-03-25T14:36:56.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 10 monclient: build_initial_monmap 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 10 monclient: monmap: 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:56.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 1 librados: starting msgr at 2024-03-25T14:36:56.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 1 librados: starting objecter 2024-03-25T14:36:56.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 1 Processor -- start 2024-03-25T14:36:56.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 1 -- start start 2024-03-25T14:36:56.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 1 librados: setting wanted keys 2024-03-25T14:36:56.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 1 librados: calling monclient init 2024-03-25T14:36:56.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.262+0000 7f393bd9d640 10 monclient: init 2024-03-25T14:36:56.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient: _reopen_session rank -1 2024-03-25T14:36:56.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:36:56.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7f39340a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7f393413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7f39341430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39341da8f0 con 0x7f39341430c0 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39341db770 con 0x7f39340a98f0 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39341dab40 con 0x7f393413f710 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient(hunting): _renew_subs 2024-03-25T14:36:56.269 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.263+0000 7f393bd9d640 10 monclient(hunting): authenticate will time out at 2101.365855s 2024-03-25T14:36:56.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.273+0000 7f3939b39640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f39340a98f0 0x7f39341db060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:38618/0 (socket says 172.21.15.26:38618) 2024-03-25T14:36:56.277 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.273+0000 7f3939b39640 1 -- 172.21.15.26:0/3502534882 learned_addr learned my addr 172.21.15.26:0/3502534882 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:36:56.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.274+0000 7f392a7fc640 10 client.?.objecter ms_handle_connect 0x7f39340a98f0 2024-03-25T14:36:56.278 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.274+0000 7f392a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:56.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.277+0000 7f392a7fc640 10 client.?.objecter ms_handle_connect 0x7f393413f710 2024-03-25T14:36:56.281 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.277+0000 7f392a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:56.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 10 client.?.objecter ms_handle_connect 0x7f39341430c0 2024-03-25T14:36:56.284 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:36:56.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706242115 0 0) 0x7f39341db770 con 0x7f39340a98f0 2024-03-25T14:36:56.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:36:56.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:56.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:56.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 10 monclient(hunting): my global_id is 4660 2024-03-25T14:36:56.285 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.281+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3918004eb0 con 0x7f39340a98f0 2024-03-25T14:36:56.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.292+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042266915 0 0) 0x7f39341dab40 con 0x7f393413f710 2024-03-25T14:36:56.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.292+0000 7f392a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:36:56.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.292+0000 7f392a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:56.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.292+0000 7f392a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:56.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.292+0000 7f392a7fc640 10 monclient(hunting): my global_id is 4622 2024-03-25T14:36:56.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.292+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39341db770 con 0x7f393413f710 2024-03-25T14:36:56.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.296+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431599429 0 0) 0x7f39341da8f0 con 0x7f39341430c0 2024-03-25T14:36:56.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.296+0000 7f392a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:36:56.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.296+0000 7f392a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:36:56.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.296+0000 7f392a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:36:56.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.296+0000 7f392a7fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T14:36:56.300 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.296+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39341dab40 con 0x7f39341430c0 2024-03-25T14:36:56.309 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 32165047 0 0) 0x7f39341db770 con 0x7f393413f710 2024-03-25T14:36:56.309 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39341da8f0 con 0x7f393413f710 2024-03-25T14:36:56.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3924001870 con 0x7f393413f710 2024-03-25T14:36:56.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:56.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:36:56.310 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 10 monclient(hunting): dump: 2024-03-25T14:36:56.310 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:56.311 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:56.312 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:56.313 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.305+0000 7f392a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.310+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145814954 0 0) 0x7f3918004eb0 con 0x7f39340a98f0 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.310+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39341db770 con 0x7f39340a98f0 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.310+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3930001d80 con 0x7f39340a98f0 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.310+0000 7f392a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.310+0000 7f392a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.310+0000 7f392a7fc640 10 monclient(hunting): dump: 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:56.314 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:56.315 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2077200267 0 0) 0x7f39341da8f0 con 0x7f393413f710 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 >> v1:172.21.15.121:6789/0 conn(0x7f39340a98f0 legacy=0x7f39341db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 >> v1:172.21.15.26:6789/0 conn(0x7f39341430c0 legacy=0x7f39341e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 1 monclient: found mon.c 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.318+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39341390e0 con 0x7f393413f710 2024-03-25T14:36:56.323 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f392a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f392a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f392a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 5 monclient: authenticate success, global_id 4622 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 10 monclient: _renew_subs 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3934137c00 con 0x7f393413f710 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 10 monclient: _renew_subs 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934138e20 con 0x7f393413f710 2024-03-25T14:36:56.324 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.319+0000 7f393bd9d640 1 librados: init done 2024-03-25T14:36:56.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.320+0000 7f393bd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:56.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.320+0000 7f393bd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:56.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.320+0000 7f393bd9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:36:56.325 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.320+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39341dacd0 con 0x7f393413f710 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3924003610 con 0x7f393413f710 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3924001d80 con 0x7f393413f710 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392a7fc640 10 monclient: dump: 2024-03-25T14:36:56.647 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:36:56.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.643+0000 7f392bfff640 4 set_mon_vals no callback set 2024-03-25T14:36:56.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.644+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f3924003b20 con 0x7f393413f710 2024-03-25T14:36:56.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.645+0000 7f3939b39640 10 monclient: get_auth_request con 0x7f391806dfa0 auth_method 0 2024-03-25T14:36:56.649 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.645+0000 7f392a7fc640 10 client.4622.objecter ms_handle_connect 0x7f391806dfa0 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 375199907 0 0) 0x7f3924053500 con 0x7f393413f710 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 10 client.4622.objecter ms_dispatch 0x7f3934000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f392404f010 con 0x7f393413f710 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:36:56.656 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.652+0000 7f392a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:36:56.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.894+0000 7f393bd9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:56.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.894+0000 7f393bd9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:56.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.894+0000 7f393bd9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:36:56.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.894+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39340627b0 con 0x7f393413f710 2024-03-25T14:36:56.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.926+0000 7f392a7fc640 1 -- 172.21.15.26:0/3502534882 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2917712005) 0x7f3924050f40 con 0x7f393413f710 2024-03-25T14:36:56.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.926+0000 7f392a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:36:56.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.926+0000 7f392a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:36:56.930 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:36:56.930 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":10,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:27:04.313759+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:27:04.313759+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:36:56.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 10 librados: watch_flush enter 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 10 librados: watch_flush exit 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 10 client.4622.objecter successfully canceled tick 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 >> v1:172.21.15.121:6800/288813066 conn(0x7f391806dfa0 legacy=0x7f3918070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 10 monclient: shutdown 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 >> v1:172.21.15.26:6790/0 conn(0x7f393413f710 legacy=0x7f3934138530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.928+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 shutdown_connections 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.929+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 >> 172.21.15.26:0/3502534882 conn(0x7f39340b7ea0 msgr2=0x7f3934074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.929+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 shutdown_connections 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.929+0000 7f393bd9d640 1 -- 172.21.15.26:0/3502534882 wait complete. 2024-03-25T14:36:56.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:36:56.929+0000 7f393bd9d640 1 librados: shutdown 2024-03-25T14:36:56.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:02.802 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:21.663 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.858 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:21.720 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.884 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:21.745 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.903 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:21.781 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.909 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:21.786 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.909 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:21.770 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.917 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:21.779 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.938 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:21.815 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:02.942 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:21.803 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:03.754 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:03.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.866+0000 7f3fe8af6640 10 monclient: get_monmap_and_config 2024-03-25T14:37:03.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.866+0000 7f3fe8af6640 10 monclient: build_initial_monmap 2024-03-25T14:37:03.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.866+0000 7f3fe8af6640 10 monclient: monmap: 2024-03-25T14:37:03.870 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:03.870 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:03.867634+0000 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:03.867634+0000 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:03.871 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.867+0000 7f3fe8af6640 1 Processor -- start 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.867+0000 7f3fe8af6640 1 -- start start 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.867+0000 7f3fe8af6640 10 monclient: init 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe40b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe40b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40af940 con 0x7f3fe414be90 2024-03-25T14:37:03.872 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40b0a70 con 0x7f3fe40b1730 2024-03-25T14:37:03.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40b0d50 con 0x7f3fe40b4d60 2024-03-25T14:37:03.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:03.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.868+0000 7f3fe8af6640 10 monclient(hunting): authenticate will time out at 2108.971303s 2024-03-25T14:37:03.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.876+0000 7f3fe259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3fe40b4d60 0x7f3fe413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58666/0 (socket says 172.21.15.26:58666) 2024-03-25T14:37:03.880 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.876+0000 7f3fe259c640 1 -- 172.21.15.26:0/2207350187 learned_addr learned my addr 172.21.15.26:0/2207350187 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:03.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.882+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519626569 0 0) 0x7f3fe40b0d50 con 0x7f3fe40b4d60 2024-03-25T14:37:03.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.882+0000 7f3fe159a640 10 monclient(hunting): _negotiate 2024-03-25T14:37:03.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.882+0000 7f3fe159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:03.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.882+0000 7f3fe159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:03.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.882+0000 7f3fe159a640 10 monclient(hunting): my global_id is 4625 2024-03-25T14:37:03.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.882+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fcc003520 con 0x7f3fe40b4d60 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565787576 0 0) 0x7f3fe40af940 con 0x7f3fe414be90 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient(hunting): _negotiate 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient(hunting): my global_id is 4677 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe40b0d50 con 0x7f3fe414be90 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3493865545 0 0) 0x7f3fcc003520 con 0x7f3fe40b4d60 2024-03-25T14:37:03.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 >> v1:172.21.15.121:6789/0 conn(0x7f3fe40b1730 legacy=0x7f3fe40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 >> v1:172.21.15.26:6789/0 conn(0x7f3fe414be90 legacy=0x7f3fe414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: _finish_hunting 0 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 monclient: found mon.noname-b 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40b1030 con 0x7f3fe40b4d60 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: _finish_auth 0 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: _check_auth_tickets 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3fd4001690 con 0x7f3fe40b4d60 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:03.901 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe159a640 10 monclient: dump: 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe8af6640 5 monclient: authenticate success, global_id 4625 2024-03-25T14:37:03.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.895+0000 7f3fe8af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fd40017f0 con 0x7f3fe40b4d60 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe159a640 1 -- 172.21.15.26:0/2207350187 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3fd4001c00 con 0x7f3fe40b4d60 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe159a640 10 monclient: dump: 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:03.912 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe8af6640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:03.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.908+0000 7f3fe8af6640 4 set_mon_vals no callback set 2024-03-25T14:37:03.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 10 monclient: shutdown 2024-03-25T14:37:03.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:03.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 1 -- 172.21.15.26:0/2207350187 >> v1:172.21.15.26:6790/0 conn(0x7f3fe40b4d60 legacy=0x7f3fe413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:03.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 1 -- 172.21.15.26:0/2207350187 shutdown_connections 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 1 -- 172.21.15.26:0/2207350187 >> 172.21.15.26:0/2207350187 conn(0x7f3fe40617a0 msgr2=0x7f3fe4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 1 -- 172.21.15.26:0/2207350187 shutdown_connections 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.909+0000 7f3fe8af6640 1 -- 172.21.15.26:0/2207350187 wait complete. 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 10 monclient: build_initial_monmap 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 10 monclient: monmap: 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:03.916 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 1 librados: starting msgr at 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 1 librados: starting objecter 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 1 Processor -- start 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 1 -- start start 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 1 librados: setting wanted keys 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 1 librados: calling monclient init 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.910+0000 7f3fe8af6640 10 monclient: init 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:37:03.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe40b1730 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe40b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe414be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe41d7980 con 0x7f3fe40b1730 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe41420a0 con 0x7f3fe40b4d60 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe41432c0 con 0x7f3fe414be90 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:03.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.911+0000 7f3fe8af6640 10 monclient(hunting): authenticate will time out at 2109.014445s 2024-03-25T14:37:03.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.920+0000 7f3fe259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3fe40b1730 0x7f3fe4141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60222/0 (socket says 172.21.15.26:60222) 2024-03-25T14:37:03.924 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.920+0000 7f3fe259c640 1 -- 172.21.15.26:0/4200831477 learned_addr learned my addr 172.21.15.26:0/4200831477 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:03.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.925+0000 7f3fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f3fe40b1730 2024-03-25T14:37:03.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.925+0000 7f3fd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:03.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.929+0000 7f3fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f3fe40b4d60 2024-03-25T14:37:03.932 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.929+0000 7f3fd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:03.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.932+0000 7f3fd2ffd640 10 client.?.objecter ms_handle_connect 0x7f3fe414be90 2024-03-25T14:37:03.936 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.932+0000 7f3fd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:03.941 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.938+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700811537 0 0) 0x7f3fe41420a0 con 0x7f3fe40b4d60 2024-03-25T14:37:03.941 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.938+0000 7f3fd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:03.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.938+0000 7f3fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:03.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.938+0000 7f3fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:03.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.938+0000 7f3fd2ffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T14:37:03.942 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.938+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc4004eb0 con 0x7f3fe40b4d60 2024-03-25T14:37:03.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.939+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007866659 0 0) 0x7f3fe41432c0 con 0x7f3fe414be90 2024-03-25T14:37:03.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.939+0000 7f3fd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:03.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.939+0000 7f3fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:03.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.939+0000 7f3fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:03.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.939+0000 7f3fd2ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T14:37:03.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.939+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe41420a0 con 0x7f3fe414be90 2024-03-25T14:37:03.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.942+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359511595 0 0) 0x7f3fe41d7980 con 0x7f3fe40b1730 2024-03-25T14:37:03.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.942+0000 7f3fd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:03.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.942+0000 7f3fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:03.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.942+0000 7f3fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:03.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.942+0000 7f3fd2ffd640 10 monclient(hunting): my global_id is 4680 2024-03-25T14:37:03.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.942+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe41432c0 con 0x7f3fe40b1730 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2720381124 0 0) 0x7f3fe41420a0 con 0x7f3fe414be90 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe41d7980 con 0x7f3fe414be90 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3fdc001b90 con 0x7f3fe414be90 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:03.964 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:03.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.960+0000 7f3fd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.964+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490803958 0 0) 0x7f3fe41432c0 con 0x7f3fe40b1730 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.964+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe41420a0 con 0x7f3fe40b1730 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.964+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3fd4001bc0 con 0x7f3fe40b1730 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.964+0000 7f3fd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.964+0000 7f3fd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.964+0000 7f3fd2ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:03.970 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:03.971 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:03.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.972+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369506039 0 0) 0x7f3fe41d7980 con 0x7f3fe414be90 2024-03-25T14:37:03.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.972+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 >> v1:172.21.15.121:6789/0 conn(0x7f3fe40b4d60 legacy=0x7f3fe4138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:03.977 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.972+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 >> v1:172.21.15.26:6789/0 conn(0x7f3fe40b1730 legacy=0x7f3fe4141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 1 monclient: found mon.c 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4143940 con 0x7f3fe414be90 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 10 monclient: _finish_auth 0 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 5 monclient: authenticate success, global_id 4628 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 10 monclient: _renew_subs 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe4138a70 con 0x7f3fe414be90 2024-03-25T14:37:03.978 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 10 monclient: _renew_subs 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe41d8700 con 0x7f3fe414be90 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.973+0000 7f3fe8af6640 1 librados: init done 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.974+0000 7f3fe8af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.974+0000 7f3fe8af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.974+0000 7f3fe8af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:03.979 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:03.974+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4143130 con 0x7f3fe414be90 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fdc002040 con 0x7f3fe414be90 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f3fdc003b10 con 0x7f3fe414be90 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fd2ffd640 10 monclient: dump: 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:04.293 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:04.294 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.289+0000 7f3fe0d99640 4 set_mon_vals no callback set 2024-03-25T14:37:04.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.290+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f3fdc019ae0 con 0x7f3fe414be90 2024-03-25T14:37:04.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.290+0000 7f3fe259c640 10 monclient: get_auth_request con 0x7f3fc406df30 auth_method 0 2024-03-25T14:37:04.295 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.291+0000 7f3fd2ffd640 10 client.4628.objecter ms_handle_connect 0x7f3fc406df30 2024-03-25T14:37:04.298 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f3fdc053740 con 0x7f3fe414be90 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 10 client.4628.objecter ms_dispatch 0x7f3fe4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fdc04f250 con 0x7f3fe414be90 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:04.299 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.295+0000 7f3fd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:04.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.540+0000 7f3fe8af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:04.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.540+0000 7f3fe8af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:04.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.540+0000 7f3fe8af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:04.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.540+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fe41401f0 con 0x7f3fe414be90 2024-03-25T14:37:04.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.572+0000 7f3fd2ffd640 1 -- 172.21.15.26:0/4200831477 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f3fdc051180 con 0x7f3fe414be90 2024-03-25T14:37:04.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.572+0000 7f3fd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:04.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.572+0000 7f3fd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:04.577 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:04.577 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 10 librados: watch_flush enter 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 10 librados: watch_flush exit 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 10 client.4628.objecter successfully canceled tick 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 >> v1:172.21.15.121:6800/288813066 conn(0x7f3fc406df30 legacy=0x7f3fc40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 10 monclient: shutdown 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 >> v1:172.21.15.26:6790/0 conn(0x7f3fe414be90 legacy=0x7f3fe41d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:04.580 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 shutdown_connections 2024-03-25T14:37:04.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 >> 172.21.15.26:0/4200831477 conn(0x7f3fe40617a0 msgr2=0x7f3fe4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:04.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 shutdown_connections 2024-03-25T14:37:04.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 -- 172.21.15.26:0/4200831477 wait complete. 2024-03-25T14:37:04.581 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:04.575+0000 7f3fe8af6640 1 librados: shutdown 2024-03-25T14:37:04.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:11.400 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:11.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.509+0000 7f5c0697b640 10 monclient: get_monmap_and_config 2024-03-25T14:37:11.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.509+0000 7f5c0697b640 10 monclient: build_initial_monmap 2024-03-25T14:37:11.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.509+0000 7f5c0697b640 10 monclient: monmap: 2024-03-25T14:37:11.514 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:11.511399+0000 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:11.511399+0000 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:11.515 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.510+0000 7f5c0697b640 1 Processor -- start 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.510+0000 7f5c0697b640 1 -- start start 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.510+0000 7f5c0697b640 10 monclient: init 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c000a96e0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c00142460 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:11.517 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c000b4d80 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:11.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000a9ac0 con 0x7f5c000b4d80 2024-03-25T14:37:11.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c001400b0 con 0x7f5c00142460 2024-03-25T14:37:11.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b1af0 con 0x7f5c000a96e0 2024-03-25T14:37:11.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:11.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.511+0000 7f5c0697b640 10 monclient(hunting): authenticate will time out at 2116.615338s 2024-03-25T14:37:11.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.516+0000 7f5c04f18640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5c000b4d80 0x7f5c0014dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59604/0 (socket says 172.21.15.26:59604) 2024-03-25T14:37:11.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.516+0000 7f5c04f18640 1 -- 172.21.15.26:0/485831737 learned_addr learned my addr 172.21.15.26:0/485831737 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:11.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.535+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706845502 0 0) 0x7f5c001400b0 con 0x7f5c00142460 2024-03-25T14:37:11.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.535+0000 7f5bfeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:11.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.535+0000 7f5bfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:11.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.535+0000 7f5bfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:11.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.535+0000 7f5bfeffd640 10 monclient(hunting): my global_id is 4669 2024-03-25T14:37:11.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.535+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be8003520 con 0x7f5c00142460 2024-03-25T14:37:11.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.545+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620461434 0 0) 0x7f5c000b1af0 con 0x7f5c000a96e0 2024-03-25T14:37:11.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.545+0000 7f5bfeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:11.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.545+0000 7f5bfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:11.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.545+0000 7f5bfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:11.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.545+0000 7f5bfeffd640 10 monclient(hunting): my global_id is 4631 2024-03-25T14:37:11.550 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.545+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c001400b0 con 0x7f5c000a96e0 2024-03-25T14:37:11.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.548+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205679293 0 0) 0x7f5c000a9ac0 con 0x7f5c000b4d80 2024-03-25T14:37:11.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.548+0000 7f5bfeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:11.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.548+0000 7f5bfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:11.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.548+0000 7f5bfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:11.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.548+0000 7f5bfeffd640 10 monclient(hunting): my global_id is 4683 2024-03-25T14:37:11.553 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.548+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000b1af0 con 0x7f5c000b4d80 2024-03-25T14:37:11.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165670044 0 0) 0x7f5be8003520 con 0x7f5c00142460 2024-03-25T14:37:11.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 >> v1:172.21.15.26:6790/0 conn(0x7f5c000a96e0 legacy=0x7f5c000b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:11.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 >> v1:172.21.15.26:6789/0 conn(0x7f5c000b4d80 legacy=0x7f5c0014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:11.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: _finish_hunting 0 2024-03-25T14:37:11.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 1 monclient: found mon.noname-c 2024-03-25T14:37:11.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c001428c0 con 0x7f5c00142460 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: _finish_auth 0 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: _check_auth_tickets 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5bf4001690 con 0x7f5c00142460 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5bfeffd640 10 monclient: dump: 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:11.556 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5c0697b640 5 monclient: authenticate success, global_id 4669 2024-03-25T14:37:11.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.550+0000 7f5c0697b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:11.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bf40017f0 con 0x7f5c00142460 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5bfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5bfeffd640 1 -- 172.21.15.26:0/485831737 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5bf4001c00 con 0x7f5c00142460 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5c0697b640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5c0697b640 4 set_mon_vals no callback set 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5bfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5bfeffd640 10 monclient: dump: 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.579 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:11.580 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:11.580 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:11.580 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:11.580 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:11.580 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5c0697b640 10 monclient: shutdown 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5c0697b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.574+0000 7f5c0697b640 1 -- 172.21.15.26:0/485831737 >> v1:172.21.15.121:6789/0 conn(0x7f5c00142460 legacy=0x7f5c000b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.575+0000 7f5c0697b640 1 -- 172.21.15.26:0/485831737 shutdown_connections 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.575+0000 7f5c0697b640 1 -- 172.21.15.26:0/485831737 >> 172.21.15.26:0/485831737 conn(0x7f5c000617a0 msgr2=0x7f5c00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.575+0000 7f5c0697b640 1 -- 172.21.15.26:0/485831737 shutdown_connections 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.575+0000 7f5c0697b640 1 -- 172.21.15.26:0/485831737 wait complete. 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.575+0000 7f5c0697b640 10 monclient: build_initial_monmap 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.575+0000 7f5c0697b640 10 monclient: monmap: 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:11.582 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.583 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 librados: starting msgr at 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 librados: starting objecter 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 Processor -- start 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 -- start start 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 librados: setting wanted keys 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 librados: calling monclient init 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient: init 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c000a96e0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c000b4d80 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:11.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c00142460 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:11.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000af070 con 0x7f5c00142460 2024-03-25T14:37:11.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000af400 con 0x7f5c000b4d80 2024-03-25T14:37:11.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0013ed70 con 0x7f5c000a96e0 2024-03-25T14:37:11.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:11.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.576+0000 7f5c0697b640 10 monclient(hunting): authenticate will time out at 2116.680586s 2024-03-25T14:37:11.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.587+0000 7f5bff7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5c000b4d80 0x7f5c0013e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:38904/0 (socket says 172.21.15.26:38904) 2024-03-25T14:37:11.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.587+0000 7f5bff7fe640 1 -- 172.21.15.26:0/886593619 learned_addr learned my addr 172.21.15.26:0/886593619 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:11.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.592+0000 7f5bfcff9640 10 client.?.objecter ms_handle_connect 0x7f5c000b4d80 2024-03-25T14:37:11.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.592+0000 7f5bfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:11.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.598+0000 7f5bfcff9640 10 client.?.objecter ms_handle_connect 0x7f5c00142460 2024-03-25T14:37:11.602 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.598+0000 7f5bfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:11.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.601+0000 7f5bfcff9640 10 client.?.objecter ms_handle_connect 0x7f5c000a96e0 2024-03-25T14:37:11.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.601+0000 7f5bfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:11.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.607+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789798918 0 0) 0x7f5c000af400 con 0x7f5c000b4d80 2024-03-25T14:37:11.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.607+0000 7f5bfcff9640 10 monclient(hunting): _negotiate 2024-03-25T14:37:11.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.607+0000 7f5bfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:11.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.607+0000 7f5bfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:11.611 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.607+0000 7f5bfcff9640 10 monclient(hunting): my global_id is 4672 2024-03-25T14:37:11.612 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.607+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bec004eb0 con 0x7f5c000b4d80 2024-03-25T14:37:11.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.612+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079561294 0 0) 0x7f5c0013ed70 con 0x7f5c000a96e0 2024-03-25T14:37:11.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.612+0000 7f5bfcff9640 10 monclient(hunting): _negotiate 2024-03-25T14:37:11.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.612+0000 7f5bfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:11.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.612+0000 7f5bfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:11.616 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.612+0000 7f5bfcff9640 10 monclient(hunting): my global_id is 4634 2024-03-25T14:37:11.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.612+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000af400 con 0x7f5c000a96e0 2024-03-25T14:37:11.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.614+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618400115 0 0) 0x7f5c000af070 con 0x7f5c00142460 2024-03-25T14:37:11.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.614+0000 7f5bfcff9640 10 monclient(hunting): _negotiate 2024-03-25T14:37:11.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.614+0000 7f5bfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:11.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.614+0000 7f5bfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:11.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.614+0000 7f5bfcff9640 10 monclient(hunting): my global_id is 4686 2024-03-25T14:37:11.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.614+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0013ed70 con 0x7f5c00142460 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092452069 0 0) 0x7f5bec004eb0 con 0x7f5c000b4d80 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c000af070 con 0x7f5c000b4d80 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5bf4001bc0 con 0x7f5c000b4d80 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 10 monclient(hunting): dump: 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:11.621 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.616+0000 7f5bfcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.623+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465950841 0 0) 0x7f5c000af400 con 0x7f5c000a96e0 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.623+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bec004eb0 con 0x7f5c000a96e0 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.623+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5bf0001970 con 0x7f5c000a96e0 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.623+0000 7f5bfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.623+0000 7f5bfcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.623+0000 7f5bfcff9640 10 monclient(hunting): dump: 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:11.628 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:11.629 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:11.629 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:11.629 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.636 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 253492343 0 0) 0x7f5bec004eb0 con 0x7f5c000a96e0 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 >> v1:172.21.15.121:6789/0 conn(0x7f5c000b4d80 legacy=0x7f5c0013e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 >> v1:172.21.15.26:6789/0 conn(0x7f5c00142460 legacy=0x7f5c001db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 10 monclient: _finish_hunting 0 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 1 monclient: found mon.c 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.631+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0013e7c0 con 0x7f5c000a96e0 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5bfcff9640 10 monclient: _finish_auth 0 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5bfcff9640 10 monclient: _check_auth_tickets 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5bfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 5 monclient: authenticate success, global_id 4634 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 10 monclient: _renew_subs 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:11.637 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0013ea10 con 0x7f5c000a96e0 2024-03-25T14:37:11.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:11.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 10 monclient: _renew_subs 2024-03-25T14:37:11.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:11.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c000acd60 con 0x7f5c000a96e0 2024-03-25T14:37:11.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.632+0000 7f5c0697b640 1 librados: init done 2024-03-25T14:37:11.638 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.633+0000 7f5c0697b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:11.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.633+0000 7f5c0697b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:11.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.633+0000 7f5c0697b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:11.639 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.633+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c000ab1b0 con 0x7f5c000a96e0 2024-03-25T14:37:11.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bf0003610 con 0x7f5c000a96e0 2024-03-25T14:37:11.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:11.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f5bf0003920 con 0x7f5c000a96e0 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfcff9640 10 monclient: dump: 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:11.950 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.944+0000 7f5bfe7fc640 4 set_mon_vals no callback set 2024-03-25T14:37:11.951 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.945+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f5bf00198f0 con 0x7f5c000a96e0 2024-03-25T14:37:11.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.946+0000 7f5bff7fe640 10 monclient: get_auth_request con 0x7f5bec06df70 auth_method 0 2024-03-25T14:37:11.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.946+0000 7f5bfcff9640 10 client.4634.objecter ms_handle_connect 0x7f5bec06df70 2024-03-25T14:37:11.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f5bf00535e0 con 0x7f5c000a96e0 2024-03-25T14:37:11.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 10 client.4634.objecter ms_dispatch 0x7f5c00000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:11.957 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:11.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:11.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T14:37:11.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bf004ef20 con 0x7f5c000a96e0 2024-03-25T14:37:11.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:11.958 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:11.952+0000 7f5bfcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:12.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.187+0000 7f5c0697b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:12.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.187+0000 7f5c0697b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:12.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.187+0000 7f5c0697b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:12.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.187+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c00143520 con 0x7f5c000a96e0 2024-03-25T14:37:12.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.228+0000 7f5bfcff9640 1 -- 172.21.15.26:0/886593619 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f5bf0051020 con 0x7f5c000a96e0 2024-03-25T14:37:12.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.228+0000 7f5bfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:12.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.228+0000 7f5bfcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:12.234 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:12.234 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 10 librados: watch_flush enter 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 10 librados: watch_flush exit 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 10 client.4634.objecter successfully canceled tick 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 >> v1:172.21.15.121:6800/288813066 conn(0x7f5bec06df70 legacy=0x7f5bec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 10 monclient: shutdown 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 >> v1:172.21.15.26:6790/0 conn(0x7f5c000a96e0 legacy=0x7f5c000ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 shutdown_connections 2024-03-25T14:37:12.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 >> 172.21.15.26:0/886593619 conn(0x7f5c000617a0 msgr2=0x7f5c000b7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:12.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.231+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 shutdown_connections 2024-03-25T14:37:12.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.232+0000 7f5c0697b640 1 -- 172.21.15.26:0/886593619 wait complete. 2024-03-25T14:37:12.237 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:12.232+0000 7f5c0697b640 1 librados: shutdown 2024-03-25T14:37:12.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:19.055 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.168+0000 7f31dbd8c640 10 monclient: get_monmap_and_config 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.168+0000 7f31dbd8c640 10 monclient: build_initial_monmap 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.168+0000 7f31dbd8c640 10 monclient: monmap: 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:19.170562+0000 2024-03-25T14:37:19.173 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:19.170562+0000 2024-03-25T14:37:19.174 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:19.174 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.174 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:19.174 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:19.174 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:19.174 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.169+0000 7f31dbd8c640 1 Processor -- start 2024-03-25T14:37:19.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.169+0000 7f31dbd8c640 1 -- start start 2024-03-25T14:37:19.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.169+0000 7f31dbd8c640 10 monclient: init 2024-03-25T14:37:19.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:19.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:37:19.175 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient(hunting): picked mon.noname-a con 0x7f31d40a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient(hunting): picked mon.noname-c con 0x7f31d413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient(hunting): picked mon.noname-b con 0x7f31d41430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d4142520 con 0x7f31d40a98f0 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40b28a0 con 0x7f31d413f710 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40b2a30 con 0x7f31d41430c0 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:19.176 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.170+0000 7f31dbd8c640 10 monclient(hunting): authenticate will time out at 2124.274131s 2024-03-25T14:37:19.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.180+0000 7f31d9327640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f31d413f710 0x7f31d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60688/0 (socket says 172.21.15.26:60688) 2024-03-25T14:37:19.185 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.180+0000 7f31d9327640 1 -- 172.21.15.26:0/201709376 learned_addr learned my addr 172.21.15.26:0/201709376 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:19.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.189+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134456515 0 0) 0x7f31d40b28a0 con 0x7f31d413f710 2024-03-25T14:37:19.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.189+0000 7f31d8b26640 10 monclient(hunting): _negotiate 2024-03-25T14:37:19.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.189+0000 7f31d8b26640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:19.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.189+0000 7f31d8b26640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:19.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.189+0000 7f31d8b26640 10 monclient(hunting): my global_id is 4675 2024-03-25T14:37:19.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.189+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c0003520 con 0x7f31d413f710 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031007100 0 0) 0x7f31c0003520 con 0x7f31d413f710 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 >> v1:172.21.15.26:6790/0 conn(0x7f31d41430c0 legacy=0x7f31d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 >> v1:172.21.15.26:6789/0 conn(0x7f31d40a98f0 legacy=0x7f31d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: _finish_hunting 0 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 1 monclient: found mon.noname-c 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40b44c0 con 0x7f31d413f710 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: _finish_auth 0 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: _check_auth_tickets 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:19.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f31c4001690 con 0x7f31d413f710 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31d8b26640 10 monclient: dump: 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:19.210 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31dbd8c640 5 monclient: authenticate success, global_id 4675 2024-03-25T14:37:19.211 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.204+0000 7f31dbd8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:19.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31c40017f0 con 0x7f31d413f710 2024-03-25T14:37:19.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31d8b26640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:19.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31d8b26640 1 -- 172.21.15.26:0/201709376 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f31c4001c40 con 0x7f31d413f710 2024-03-25T14:37:19.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31d8b26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:19.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31d8b26640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31d8b26640 10 monclient: dump: 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:19.218 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 4 set_mon_vals no callback set 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 10 monclient: shutdown 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 1 -- 172.21.15.26:0/201709376 >> v1:172.21.15.121:6789/0 conn(0x7f31d413f710 legacy=0x7f31d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 1 -- 172.21.15.26:0/201709376 shutdown_connections 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 1 -- 172.21.15.26:0/201709376 >> 172.21.15.26:0/201709376 conn(0x7f31d40b7ea0 msgr2=0x7f31d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.212+0000 7f31dbd8c640 1 -- 172.21.15.26:0/201709376 shutdown_connections 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.213+0000 7f31dbd8c640 1 -- 172.21.15.26:0/201709376 wait complete. 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.213+0000 7f31dbd8c640 10 monclient: build_initial_monmap 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.213+0000 7f31dbd8c640 10 monclient: monmap: 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:19.220 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 1 librados: starting msgr at 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 1 librados: starting objecter 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 1 Processor -- start 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 1 -- start start 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 1 librados: setting wanted keys 2024-03-25T14:37:19.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 1 librados: calling monclient init 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 10 monclient: init 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 10 monclient(hunting): picked mon.noname-c con 0x7f31d40a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.214+0000 7f31dbd8c640 10 monclient(hunting): picked mon.noname-a con 0x7f31d413f710 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.215+0000 7f31dbd8c640 10 monclient(hunting): picked mon.noname-b con 0x7f31d41430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.215+0000 7f31dbd8c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d4136cc0 con 0x7f31d413f710 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.215+0000 7f31dbd8c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d41dc070 con 0x7f31d41430c0 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.215+0000 7f31dbd8c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40b77c0 con 0x7f31d40a98f0 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.215+0000 7f31dbd8c640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:19.222 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.215+0000 7f31dbd8c640 10 monclient(hunting): authenticate will time out at 2124.318644s 2024-03-25T14:37:19.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.218+0000 7f31d9b28640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f31d40a98f0 0x7f31d4137630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:49718/0 (socket says 172.21.15.26:49718) 2024-03-25T14:37:19.223 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.218+0000 7f31d9b28640 1 -- 172.21.15.26:0/2714875658 learned_addr learned my addr 172.21.15.26:0/2714875658 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:19.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.221+0000 7f31ca7fc640 10 client.?.objecter ms_handle_connect 0x7f31d40a98f0 2024-03-25T14:37:19.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.221+0000 7f31ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:19.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.224+0000 7f31ca7fc640 10 client.?.objecter ms_handle_connect 0x7f31d41430c0 2024-03-25T14:37:19.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.224+0000 7f31ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:19.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.226+0000 7f31ca7fc640 10 client.?.objecter ms_handle_connect 0x7f31d413f710 2024-03-25T14:37:19.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.226+0000 7f31ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:19.235 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.231+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169618606 0 0) 0x7f31d40b77c0 con 0x7f31d40a98f0 2024-03-25T14:37:19.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.231+0000 7f31ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:37:19.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.231+0000 7f31ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:19.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.231+0000 7f31ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:19.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.231+0000 7f31ca7fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T14:37:19.236 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.231+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b8004eb0 con 0x7f31d40a98f0 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572058594 0 0) 0x7f31d41dc070 con 0x7f31d41430c0 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): my global_id is 4678 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d40b77c0 con 0x7f31d41430c0 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424995553 0 0) 0x7f31d4136cc0 con 0x7f31d413f710 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:19.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 10 monclient(hunting): my global_id is 4692 2024-03-25T14:37:19.247 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.241+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d41dc070 con 0x7f31d413f710 2024-03-25T14:37:19.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590985115 0 0) 0x7f31b8004eb0 con 0x7f31d40a98f0 2024-03-25T14:37:19.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31d4136cc0 con 0x7f31d40a98f0 2024-03-25T14:37:19.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f31d00019d0 con 0x7f31d40a98f0 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 10 monclient(hunting): dump: 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:19.257 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:19.258 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.252+0000 7f31ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:19.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.258+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729620600 0 0) 0x7f31d41dc070 con 0x7f31d413f710 2024-03-25T14:37:19.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.258+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31b8004eb0 con 0x7f31d413f710 2024-03-25T14:37:19.262 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.258+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f31c4001b00 con 0x7f31d413f710 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.258+0000 7f31ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.258+0000 7f31ca7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.258+0000 7f31ca7fc640 10 monclient(hunting): dump: 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:19.263 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.259+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103242769 0 0) 0x7f31d40b77c0 con 0x7f31d41430c0 2024-03-25T14:37:19.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.259+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31d41dc070 con 0x7f31d41430c0 2024-03-25T14:37:19.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.259+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f31cc001b10 con 0x7f31d41430c0 2024-03-25T14:37:19.264 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.259+0000 7f31ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.259+0000 7f31ca7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.259+0000 7f31ca7fc640 10 monclient(hunting): dump: 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:19.265 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 278615290 0 0) 0x7f31d4136cc0 con 0x7f31d40a98f0 2024-03-25T14:37:19.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 >> v1:172.21.15.121:6789/0 conn(0x7f31d41430c0 legacy=0x7f31d41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.270 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 >> v1:172.21.15.26:6789/0 conn(0x7f31d413f710 legacy=0x7f31d41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 1 monclient: found mon.c 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4136f30 con 0x7f31d40a98f0 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 10 monclient: _finish_auth 0 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 5 monclient: authenticate success, global_id 4640 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 10 monclient: _renew_subs 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:19.271 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d4137210 con 0x7f31d40a98f0 2024-03-25T14:37:19.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:19.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 10 monclient: _renew_subs 2024-03-25T14:37:19.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:19.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d4135f90 con 0x7f31d40a98f0 2024-03-25T14:37:19.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.265+0000 7f31dbd8c640 1 librados: init done 2024-03-25T14:37:19.272 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.266+0000 7f31dbd8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:19.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.266+0000 7f31dbd8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:19.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.266+0000 7f31dbd8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:19.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.266+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d40627b0 con 0x7f31d40a98f0 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31d0003610 con 0x7f31d40a98f0 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f31d0003a20 con 0x7f31d40a98f0 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 10 monclient: dump: 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:19.591 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:19.592 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:19.592 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:19.592 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:19.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31cbfff640 4 set_mon_vals no callback set 2024-03-25T14:37:19.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.586+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f31d0003c40 con 0x7f31d40a98f0 2024-03-25T14:37:19.592 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.587+0000 7f31d9327640 10 monclient: get_auth_request con 0x7f31b806e0f0 auth_method 0 2024-03-25T14:37:19.593 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.587+0000 7f31ca7fc640 10 client.4640.objecter ms_handle_connect 0x7f31b806e0f0 2024-03-25T14:37:19.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.593+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f31d0054820 con 0x7f31d40a98f0 2024-03-25T14:37:19.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 10 client.4640.objecter ms_dispatch 0x7f31d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:19.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 3 client.4640.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:19.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 3 client.4640.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:19.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T14:37:19.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31d0050330 con 0x7f31d40a98f0 2024-03-25T14:37:19.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:19.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.594+0000 7f31ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:19.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.844+0000 7f31dbd8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:19.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.844+0000 7f31dbd8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:19.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.844+0000 7f31dbd8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:19.849 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.844+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31d413df30 con 0x7f31d40a98f0 2024-03-25T14:37:19.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.866+0000 7f31ca7fc640 1 -- 172.21.15.26:0/2714875658 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f31d0052260 con 0x7f31d40a98f0 2024-03-25T14:37:19.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.866+0000 7f31ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:19.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.866+0000 7f31ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:19.871 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:19.871 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:19.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.868+0000 7f31dbd8c640 10 librados: watch_flush enter 2024-03-25T14:37:19.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.868+0000 7f31dbd8c640 10 librados: watch_flush exit 2024-03-25T14:37:19.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.868+0000 7f31dbd8c640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.868+0000 7f31dbd8c640 10 client.4640.objecter successfully canceled tick 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 >> v1:172.21.15.121:6800/288813066 conn(0x7f31b806e0f0 legacy=0x7f31b80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 10 monclient: shutdown 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 >> v1:172.21.15.26:6790/0 conn(0x7f31d40a98f0 legacy=0x7f31d4137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 shutdown_connections 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 >> 172.21.15.26:0/2714875658 conn(0x7f31d40b7ea0 msgr2=0x7f31d413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 shutdown_connections 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 -- 172.21.15.26:0/2714875658 wait complete. 2024-03-25T14:37:19.875 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:19.869+0000 7f31dbd8c640 1 librados: shutdown 2024-03-25T14:37:19.886 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:22.736 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:41.598 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.748 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:41.610 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.776 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:10:41.638 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.796 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:41.674 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.800 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:41.678 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.801 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:41.663 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.810 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:41.672 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.835 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:10:41.712 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:22.850 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:10:41.712 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:26.693 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:26.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.806+0000 7f965830e640 10 monclient: get_monmap_and_config 2024-03-25T14:37:26.809 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.806+0000 7f965830e640 10 monclient: build_initial_monmap 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.806+0000 7f965830e640 10 monclient: monmap: 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:26.806998+0000 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:26.806998+0000 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:26.810 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.806+0000 7f965830e640 1 Processor -- start 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.807+0000 7f965830e640 1 -- start start 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.807+0000 7f965830e640 10 monclient: init 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient(hunting): picked mon.noname-c con 0x7f96500b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient(hunting): picked mon.noname-b con 0x7f96500b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient(hunting): picked mon.noname-a con 0x7f965014be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500af940 con 0x7f965014be90 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500b0a70 con 0x7f96500b4d60 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500b0d50 con 0x7f96500b1730 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.808+0000 7f965830e640 10 monclient(hunting): authenticate will time out at 2131.910893s 2024-03-25T14:37:26.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.817+0000 7f96558a9640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f96500b1730 0x7f96500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54162/0 (socket says 172.21.15.26:54162) 2024-03-25T14:37:26.820 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.817+0000 7f96558a9640 1 -- 172.21.15.26:0/756308153 learned_addr learned my addr 172.21.15.26:0/756308153 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:26.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.827+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674069854 0 0) 0x7f96500b0d50 con 0x7f96500b1730 2024-03-25T14:37:26.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.827+0000 7f96550a8640 10 monclient(hunting): _negotiate 2024-03-25T14:37:26.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.827+0000 7f96550a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:26.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.827+0000 7f96550a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:26.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.827+0000 7f96550a8640 10 monclient(hunting): my global_id is 4643 2024-03-25T14:37:26.831 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.827+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c003520 con 0x7f96500b1730 2024-03-25T14:37:26.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.843+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796409092 0 0) 0x7f96500b0a70 con 0x7f96500b4d60 2024-03-25T14:37:26.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.843+0000 7f96550a8640 10 monclient(hunting): _negotiate 2024-03-25T14:37:26.846 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.843+0000 7f96550a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:26.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.843+0000 7f96550a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:26.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.843+0000 7f96550a8640 10 monclient(hunting): my global_id is 4681 2024-03-25T14:37:26.847 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.843+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96500b0d50 con 0x7f96500b4d60 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892299952 0 0) 0x7f963c003520 con 0x7f96500b1730 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 >> v1:172.21.15.121:6789/0 conn(0x7f96500b4d60 legacy=0x7f965013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 >> v1:172.21.15.26:6789/0 conn(0x7f965014be90 legacy=0x7f965014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: _finish_hunting 0 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 1 monclient: found mon.noname-b 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:37:26.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96500b1030 con 0x7f96500b1730 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: _finish_auth 0 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: _check_auth_tickets 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9640001690 con 0x7f96500b1730 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f96550a8640 10 monclient: dump: 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:26.851 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:26.852 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:26.852 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.852 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f965830e640 5 monclient: authenticate success, global_id 4643 2024-03-25T14:37:26.852 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.846+0000 7f965830e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96400017f0 con 0x7f96500b1730 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f96550a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f96550a8640 1 -- 172.21.15.26:0/756308153 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9640001c00 con 0x7f96500b1730 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f96550a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f96550a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f96550a8640 10 monclient: dump: 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:26.865 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f965830e640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:26.866 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.861+0000 7f965830e640 4 set_mon_vals no callback set 2024-03-25T14:37:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.862+0000 7f965830e640 10 monclient: shutdown 2024-03-25T14:37:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.862+0000 7f965830e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:26.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.862+0000 7f965830e640 1 -- 172.21.15.26:0/756308153 >> v1:172.21.15.26:6790/0 conn(0x7f96500b1730 legacy=0x7f96500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.862+0000 7f965830e640 1 -- 172.21.15.26:0/756308153 shutdown_connections 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.862+0000 7f965830e640 1 -- 172.21.15.26:0/756308153 >> 172.21.15.26:0/756308153 conn(0x7f96500617a0 msgr2=0x7f9650061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.863+0000 7f965830e640 1 -- 172.21.15.26:0/756308153 shutdown_connections 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.863+0000 7f965830e640 1 -- 172.21.15.26:0/756308153 wait complete. 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.863+0000 7f965830e640 10 monclient: build_initial_monmap 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.863+0000 7f965830e640 10 monclient: monmap: 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:26.869 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 1 librados: starting msgr at 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 1 librados: starting objecter 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 1 Processor -- start 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 1 -- start start 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 1 librados: setting wanted keys 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 1 librados: calling monclient init 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 10 monclient: init 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:37:26.870 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 10 monclient(hunting): picked mon.noname-c con 0x7f96500b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.864+0000 7f965830e640 10 monclient(hunting): picked mon.noname-a con 0x7f96500b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.865+0000 7f965830e640 10 monclient(hunting): picked mon.noname-b con 0x7f965014be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.865+0000 7f965830e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9650140380 con 0x7f96500b4d60 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.865+0000 7f965830e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96501423c0 con 0x7f965014be90 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.865+0000 7f965830e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96501426a0 con 0x7f96500b1730 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.865+0000 7f965830e640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:26.871 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.865+0000 7f965830e640 10 monclient(hunting): authenticate will time out at 2131.967754s 2024-03-25T14:37:26.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.880+0000 7f96568ab640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f965014be90 0x7f96501340f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:37038/0 (socket says 172.21.15.26:37038) 2024-03-25T14:37:26.883 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.880+0000 7f96568ab640 1 -- 172.21.15.26:0/3526756608 learned_addr learned my addr 172.21.15.26:0/3526756608 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:26.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.881+0000 7f9646ffd640 10 client.?.objecter ms_handle_connect 0x7f965014be90 2024-03-25T14:37:26.884 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.881+0000 7f9646ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:26.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.882+0000 7f9646ffd640 10 client.?.objecter ms_handle_connect 0x7f96500b4d60 2024-03-25T14:37:26.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.882+0000 7f9646ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:26.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.884+0000 7f9646ffd640 10 client.?.objecter ms_handle_connect 0x7f96500b1730 2024-03-25T14:37:26.887 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.884+0000 7f9646ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:26.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.894+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459791151 0 0) 0x7f96501426a0 con 0x7f96500b1730 2024-03-25T14:37:26.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.894+0000 7f9646ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:26.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.894+0000 7f9646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:26.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.894+0000 7f9646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:26.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.894+0000 7f9646ffd640 10 monclient(hunting): my global_id is 4646 2024-03-25T14:37:26.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.894+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9634004eb0 con 0x7f96500b1730 2024-03-25T14:37:26.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.896+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322365758 0 0) 0x7f96501423c0 con 0x7f965014be90 2024-03-25T14:37:26.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.896+0000 7f9646ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:26.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.896+0000 7f9646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:26.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.896+0000 7f9646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:26.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.896+0000 7f9646ffd640 10 monclient(hunting): my global_id is 4684 2024-03-25T14:37:26.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.896+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96501426a0 con 0x7f965014be90 2024-03-25T14:37:26.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.901+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572622363 0 0) 0x7f9650140380 con 0x7f96500b4d60 2024-03-25T14:37:26.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.901+0000 7f9646ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:26.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.901+0000 7f9646ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:26.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.901+0000 7f9646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:26.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.901+0000 7f9646ffd640 10 monclient(hunting): my global_id is 4698 2024-03-25T14:37:26.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.901+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96501423c0 con 0x7f96500b4d60 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042888820 0 0) 0x7f96501426a0 con 0x7f965014be90 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9650140380 con 0x7f965014be90 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9648001c10 con 0x7f965014be90 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.915 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.916 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.911+0000 7f9646ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.918+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2899736161 0 0) 0x7f9634004eb0 con 0x7f96500b1730 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.918+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96501426a0 con 0x7f96500b1730 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.918+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f964c001930 con 0x7f96500b1730 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.918+0000 7f9646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.918+0000 7f9646ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.918+0000 7f9646ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.922 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:26.923 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.930+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890025006 0 0) 0x7f96501423c0 con 0x7f96500b4d60 2024-03-25T14:37:26.933 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.930+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9634004eb0 con 0x7f96500b4d60 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.930+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f9640001b00 con 0x7f96500b4d60 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.930+0000 7f9646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.930+0000 7f9646ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.930+0000 7f9646ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:26.934 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:26.935 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4075162247 0 0) 0x7f96501426a0 con 0x7f96500b1730 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 >> v1:172.21.15.121:6789/0 conn(0x7f965014be90 legacy=0x7f96501340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 >> v1:172.21.15.26:6789/0 conn(0x7f96500b4d60 legacy=0x7f9650142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 1 monclient: found mon.c 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:26.937 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650140810 con 0x7f96500b1730 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 10 monclient: _finish_auth 0 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f9646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 5 monclient: authenticate success, global_id 4646 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 10 monclient: _renew_subs 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9650138180 con 0x7f96500b1730 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 10 monclient: _renew_subs 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96501433b0 con 0x7f96500b1730 2024-03-25T14:37:26.938 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.933+0000 7f965830e640 1 librados: init done 2024-03-25T14:37:26.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.935+0000 7f965830e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:26.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.935+0000 7f965830e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:26.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.935+0000 7f965830e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:26.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.935+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96500627b0 con 0x7f96500b1730 2024-03-25T14:37:26.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f964c003610 con 0x7f96500b1730 2024-03-25T14:37:26.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f9646ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:26.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f964c003920 con 0x7f96500b1730 2024-03-25T14:37:26.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f9646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:26.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f9646ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:26.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f9646ffd640 10 monclient: dump: 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:26.949 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:26.945+0000 7f96548a7640 4 set_mon_vals no callback set 2024-03-25T14:37:27.265 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.262+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f964c0198f0 con 0x7f96500b1730 2024-03-25T14:37:27.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.263+0000 7f96558a9640 10 monclient: get_auth_request con 0x7f963406e160 auth_method 0 2024-03-25T14:37:27.266 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.263+0000 7f9646ffd640 10 client.4646.objecter ms_handle_connect 0x7f963406e160 2024-03-25T14:37:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f964c0535c0 con 0x7f96500b1730 2024-03-25T14:37:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 10 client.4646.objecter ms_dispatch 0x7f9650000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-25T14:37:27.273 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f964c04ef00 con 0x7f96500b1730 2024-03-25T14:37:27.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:27.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.269+0000 7f9646ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:27.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.538+0000 7f965830e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:27.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.538+0000 7f965830e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:27.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.538+0000 7f965830e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:27.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.538+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96501409a0 con 0x7f96500b1730 2024-03-25T14:37:27.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.564+0000 7f9646ffd640 1 -- 172.21.15.26:0/3526756608 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f96501409a0 con 0x7f96500b1730 2024-03-25T14:37:27.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.564+0000 7f9646ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:27.568 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.564+0000 7f9646ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:27.568 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:27.568 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:27.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 10 librados: watch_flush enter 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 10 librados: watch_flush exit 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 10 client.4646.objecter successfully canceled tick 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 >> v1:172.21.15.121:6800/288813066 conn(0x7f963406e160 legacy=0x7f9634070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 10 monclient: shutdown 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 >> v1:172.21.15.26:6790/0 conn(0x7f96500b1730 legacy=0x7f9650140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 shutdown_connections 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 >> 172.21.15.26:0/3526756608 conn(0x7f96500617a0 msgr2=0x7f965013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 shutdown_connections 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.566+0000 7f965830e640 1 -- 172.21.15.26:0/3526756608 wait complete. 2024-03-25T14:37:27.571 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:27.567+0000 7f965830e640 1 librados: shutdown 2024-03-25T14:37:27.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:34.389 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.507+0000 7f273c4c9640 10 monclient: get_monmap_and_config 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.507+0000 7f273c4c9640 10 monclient: build_initial_monmap 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.507+0000 7f273c4c9640 10 monclient: monmap: 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:34.508713+0000 2024-03-25T14:37:34.511 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:34.508713+0000 2024-03-25T14:37:34.512 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:34.512 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:34.512 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:34.512 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:34.512 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:34.512 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.508+0000 7f273c4c9640 1 Processor -- start 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.508+0000 7f273c4c9640 1 -- start start 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.508+0000 7f273c4c9640 10 monclient: init 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f27340b4d60 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f27340b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f273414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27340af940 con 0x7f273414be90 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27340b0a70 con 0x7f27340b1730 2024-03-25T14:37:34.514 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27340b0d50 con 0x7f27340b4d60 2024-03-25T14:37:34.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:34.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.509+0000 7f273c4c9640 10 monclient(hunting): authenticate will time out at 2139.612427s 2024-03-25T14:37:34.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.511+0000 7f273a265640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f27340b4d60 0x7f273413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:37190/0 (socket says 172.21.15.26:37190) 2024-03-25T14:37:34.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.511+0000 7f273a265640 1 -- 172.21.15.26:0/3236024393 learned_addr learned my addr 172.21.15.26:0/3236024393 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:34.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.521+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860959372 0 0) 0x7f27340b0d50 con 0x7f27340b4d60 2024-03-25T14:37:34.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.521+0000 7f2739263640 10 monclient(hunting): _negotiate 2024-03-25T14:37:34.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.521+0000 7f2739263640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:34.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.521+0000 7f2739263640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:34.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.521+0000 7f2739263640 10 monclient(hunting): my global_id is 4649 2024-03-25T14:37:34.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.521+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2720003520 con 0x7f27340b4d60 2024-03-25T14:37:34.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 38829603 0 0) 0x7f2720003520 con 0x7f27340b4d60 2024-03-25T14:37:34.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 >> v1:172.21.15.121:6789/0 conn(0x7f27340b1730 legacy=0x7f27340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:34.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 >> v1:172.21.15.26:6789/0 conn(0x7f273414be90 legacy=0x7f273414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:34.538 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: _finish_hunting 0 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 1 monclient: found mon.noname-b 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340b1030 con 0x7f27340b4d60 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: _finish_auth 0 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: _check_auth_tickets 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2730001690 con 0x7f27340b4d60 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:34.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f2739263640 10 monclient: dump: 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f273c4c9640 5 monclient: authenticate success, global_id 4649 2024-03-25T14:37:34.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.534+0000 7f273c4c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:34.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.537+0000 7f2739263640 1 -- 172.21.15.26:0/3236024393 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27300017f0 con 0x7f27340b4d60 2024-03-25T14:37:34.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.537+0000 7f2739263640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:34.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.537+0000 7f273c4c9640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:34.541 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.537+0000 7f273c4c9640 4 set_mon_vals no callback set 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 10 monclient: shutdown 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 1 -- 172.21.15.26:0/3236024393 >> v1:172.21.15.26:6790/0 conn(0x7f27340b4d60 legacy=0x7f273413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 1 -- 172.21.15.26:0/3236024393 shutdown_connections 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 1 -- 172.21.15.26:0/3236024393 >> 172.21.15.26:0/3236024393 conn(0x7f27340617a0 msgr2=0x7f2734061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 1 -- 172.21.15.26:0/3236024393 shutdown_connections 2024-03-25T14:37:34.542 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.538+0000 7f273c4c9640 1 -- 172.21.15.26:0/3236024393 wait complete. 2024-03-25T14:37:34.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.539+0000 7f273c4c9640 10 monclient: build_initial_monmap 2024-03-25T14:37:34.544 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.539+0000 7f273c4c9640 10 monclient: monmap: 2024-03-25T14:37:34.544 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:34.544 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:34.544 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.539+0000 7f273c4c9640 1 librados: starting msgr at 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.539+0000 7f273c4c9640 1 librados: starting objecter 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.539+0000 7f273c4c9640 1 Processor -- start 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 1 -- start start 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 1 librados: setting wanted keys 2024-03-25T14:37:34.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 1 librados: calling monclient init 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient: init 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f27340b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f27340b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f273414be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2734136090 con 0x7f273414be90 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27341dad30 con 0x7f27340b4d60 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27341db960 con 0x7f27340b1730 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:34.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.540+0000 7f273c4c9640 10 monclient(hunting): authenticate will time out at 2139.643185s 2024-03-25T14:37:34.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.553+0000 7f2739a64640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f27340b4d60 0x7f273412fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39758/0 (socket says 172.21.15.26:39758) 2024-03-25T14:37:34.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.553+0000 7f2739a64640 1 -- 172.21.15.26:0/30207957 learned_addr learned my addr 172.21.15.26:0/30207957 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:34.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.558+0000 7f272affd640 10 client.?.objecter ms_handle_connect 0x7f27340b4d60 2024-03-25T14:37:34.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.558+0000 7f272affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:34.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.562+0000 7f272affd640 10 client.?.objecter ms_handle_connect 0x7f273414be90 2024-03-25T14:37:34.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.562+0000 7f272affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:34.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.563+0000 7f272affd640 10 client.?.objecter ms_handle_connect 0x7f27340b1730 2024-03-25T14:37:34.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.563+0000 7f272affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:34.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.570+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010612703 0 0) 0x7f27341db960 con 0x7f27340b1730 2024-03-25T14:37:34.573 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.570+0000 7f272affd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:34.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.570+0000 7f272affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:34.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.570+0000 7f272affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:34.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.570+0000 7f272affd640 10 monclient(hunting): my global_id is 4652 2024-03-25T14:37:34.574 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.570+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2718004eb0 con 0x7f27340b1730 2024-03-25T14:37:34.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.572+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703936163 0 0) 0x7f2734136090 con 0x7f273414be90 2024-03-25T14:37:34.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:34.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:34.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:34.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): my global_id is 4704 2024-03-25T14:37:34.576 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27341db960 con 0x7f273414be90 2024-03-25T14:37:34.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399711871 0 0) 0x7f27341dad30 con 0x7f27340b4d60 2024-03-25T14:37:34.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:34.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:34.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:34.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 10 monclient(hunting): my global_id is 4690 2024-03-25T14:37:34.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.573+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2734136090 con 0x7f27340b4d60 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3885970259 0 0) 0x7f2718004eb0 con 0x7f27340b1730 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27341dad30 con 0x7f27340b1730 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2730001ca0 con 0x7f27340b1730 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 10 monclient(hunting): dump: 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.595 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.596 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.591+0000 7f272affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:34.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.592+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1322507650 0 0) 0x7f2734136090 con 0x7f27340b4d60 2024-03-25T14:37:34.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.592+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2718004eb0 con 0x7f27340b4d60 2024-03-25T14:37:34.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.592+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2724001970 con 0x7f27340b4d60 2024-03-25T14:37:34.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.592+0000 7f272affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:34.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.592+0000 7f272affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:34.597 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.592+0000 7f272affd640 10 monclient(hunting): dump: 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.593+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2136564819 0 0) 0x7f27341db960 con 0x7f273414be90 2024-03-25T14:37:34.598 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.593+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2734136090 con 0x7f273414be90 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.593+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f272c001b10 con 0x7f273414be90 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.593+0000 7f272affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.593+0000 7f272affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.593+0000 7f272affd640 10 monclient(hunting): dump: 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:34.599 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4267488958 0 0) 0x7f27341dad30 con 0x7f27340b1730 2024-03-25T14:37:34.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 >> v1:172.21.15.121:6789/0 conn(0x7f27340b4d60 legacy=0x7f273412fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:34.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 >> v1:172.21.15.26:6789/0 conn(0x7f273414be90 legacy=0x7f2734136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:34.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 10 monclient: _finish_hunting 0 2024-03-25T14:37:34.604 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 1 monclient: found mon.c 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734130570 con 0x7f27340b1730 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 10 monclient: _finish_auth 0 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 10 monclient: _check_auth_tickets 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f272affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 5 monclient: authenticate success, global_id 4652 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 10 monclient: _renew_subs 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27341307c0 con 0x7f27340b1730 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 10 monclient: _renew_subs 2024-03-25T14:37:34.605 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:34.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2734143ae0 con 0x7f27340b1730 2024-03-25T14:37:34.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.600+0000 7f273c4c9640 1 librados: init done 2024-03-25T14:37:34.606 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.601+0000 7f273c4c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:34.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.601+0000 7f273c4c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:34.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.601+0000 7f273c4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:34.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.601+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27341dab20 con 0x7f27340b1730 2024-03-25T14:37:34.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27300011c0 con 0x7f27340b1730 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f272affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2730004cb0 con 0x7f27340b1730 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f272affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f272affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f272affd640 10 monclient: dump: 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:34.920 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.916+0000 7f2738a62640 4 set_mon_vals no callback set 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.917+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f273001ac80 con 0x7f27340b1730 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.918+0000 7f2739a64640 10 monclient: get_auth_request con 0x7f271806e0f0 auth_method 0 2024-03-25T14:37:34.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.918+0000 7f272affd640 10 client.4652.objecter ms_handle_connect 0x7f271806e0f0 2024-03-25T14:37:34.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f2730054840 con 0x7f27340b1730 2024-03-25T14:37:34.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 10 client.4652.objecter ms_dispatch 0x7f2734000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:34.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:34.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:34.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T14:37:34.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2730050350 con 0x7f27340b1730 2024-03-25T14:37:34.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:34.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:34.925+0000 7f272affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:35.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.205+0000 7f273c4c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:35.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.205+0000 7f273c4c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:35.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.205+0000 7f273c4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:37:35.209 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.205+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27340627b0 con 0x7f27340b1730 2024-03-25T14:37:35.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.246+0000 7f272affd640 1 -- 172.21.15.26:0/30207957 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f2730052280 con 0x7f27340b1730 2024-03-25T14:37:35.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.246+0000 7f272affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:35.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.246+0000 7f272affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:35.250 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:35.251 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:35.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 10 librados: watch_flush enter 2024-03-25T14:37:35.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 10 librados: watch_flush exit 2024-03-25T14:37:35.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T14:37:35.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 10 client.4652.objecter successfully canceled tick 2024-03-25T14:37:35.253 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 >> v1:172.21.15.121:6800/288813066 conn(0x7f271806e0f0 legacy=0x7f27180705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:35.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 10 monclient: shutdown 2024-03-25T14:37:35.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:35.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.249+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 >> v1:172.21.15.26:6790/0 conn(0x7f27340b1730 legacy=0x7f27341db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:35.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.250+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 shutdown_connections 2024-03-25T14:37:35.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.250+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 >> 172.21.15.26:0/30207957 conn(0x7f27340617a0 msgr2=0x7f27340b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:35.254 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.251+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 shutdown_connections 2024-03-25T14:37:35.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.251+0000 7f273c4c9640 1 -- 172.21.15.26:0/30207957 wait complete. 2024-03-25T14:37:35.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:35.251+0000 7f273c4c9640 1 librados: shutdown 2024-03-25T14:37:35.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:42.074 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:42.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.188+0000 7f2a4cfa8640 10 monclient: get_monmap_and_config 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.188+0000 7f2a4cfa8640 10 monclient: build_initial_monmap 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.188+0000 7f2a4cfa8640 10 monclient: monmap: 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:42.189102+0000 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:42.189102+0000 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:42.192 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:42.193 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.189+0000 7f2a4cfa8640 1 Processor -- start 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.189+0000 7f2a4cfa8640 1 -- start start 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.189+0000 7f2a4cfa8640 10 monclient: init 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a480a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a4813f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:42.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a481430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:42.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a48142520 con 0x7f2a481430c0 2024-03-25T14:37:42.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480b28a0 con 0x7f2a480a98f0 2024-03-25T14:37:42.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480b2a30 con 0x7f2a4813f710 2024-03-25T14:37:42.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:42.195 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.190+0000 7f2a4cfa8640 10 monclient(hunting): authenticate will time out at 2147.292862s 2024-03-25T14:37:42.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.204+0000 7f2a4659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2a480a98f0 0x7f2a480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:53298/0 (socket says 172.21.15.26:53298) 2024-03-25T14:37:42.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.204+0000 7f2a4659c640 1 -- 172.21.15.26:0/1725830574 learned_addr learned my addr 172.21.15.26:0/1725830574 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:42.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.227+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196216978 0 0) 0x7f2a480b28a0 con 0x7f2a480a98f0 2024-03-25T14:37:42.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.227+0000 7f2a4559a640 10 monclient(hunting): _negotiate 2024-03-25T14:37:42.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.227+0000 7f2a4559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:42.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.227+0000 7f2a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:42.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.227+0000 7f2a4559a640 10 monclient(hunting): my global_id is 4693 2024-03-25T14:37:42.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.227+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a30003520 con 0x7f2a480a98f0 2024-03-25T14:37:42.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.229+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471847920 0 0) 0x7f2a480b2a30 con 0x7f2a4813f710 2024-03-25T14:37:42.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.229+0000 7f2a4559a640 10 monclient(hunting): _negotiate 2024-03-25T14:37:42.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.229+0000 7f2a4559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:42.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.229+0000 7f2a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:42.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.229+0000 7f2a4559a640 10 monclient(hunting): my global_id is 4655 2024-03-25T14:37:42.233 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.229+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a480b28a0 con 0x7f2a4813f710 2024-03-25T14:37:42.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.242+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414244087 0 0) 0x7f2a48142520 con 0x7f2a481430c0 2024-03-25T14:37:42.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.243+0000 7f2a4559a640 10 monclient(hunting): _negotiate 2024-03-25T14:37:42.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.243+0000 7f2a4559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:42.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.243+0000 7f2a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:42.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.243+0000 7f2a4559a640 10 monclient(hunting): my global_id is 4707 2024-03-25T14:37:42.246 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.243+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a480b2a30 con 0x7f2a481430c0 2024-03-25T14:37:42.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554620467 0 0) 0x7f2a30003520 con 0x7f2a480a98f0 2024-03-25T14:37:42.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 >> v1:172.21.15.26:6790/0 conn(0x7f2a4813f710 legacy=0x7f2a481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 >> v1:172.21.15.26:6789/0 conn(0x7f2a481430c0 legacy=0x7f2a4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: _finish_hunting 0 2024-03-25T14:37:42.249 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 1 monclient: found mon.noname-c 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a480b44c0 con 0x7f2a480a98f0 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: _finish_auth 0 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: _check_auth_tickets 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a38001690 con 0x7f2a480a98f0 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4559a640 10 monclient: dump: 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.250 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4cfa8640 5 monclient: authenticate success, global_id 4693 2024-03-25T14:37:42.251 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.245+0000 7f2a4cfa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a380017f0 con 0x7f2a480a98f0 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4559a640 1 -- 172.21.15.26:0/1725830574 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a38001c00 con 0x7f2a480a98f0 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4559a640 10 monclient: dump: 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.255 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4cfa8640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:42.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.251+0000 7f2a4cfa8640 4 set_mon_vals no callback set 2024-03-25T14:37:42.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 10 monclient: shutdown 2024-03-25T14:37:42.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:42.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/1725830574 >> v1:172.21.15.121:6789/0 conn(0x7f2a480a98f0 legacy=0x7f2a480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/1725830574 shutdown_connections 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/1725830574 >> 172.21.15.26:0/1725830574 conn(0x7f2a480b7ea0 msgr2=0x7f2a480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/1725830574 shutdown_connections 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.252+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/1725830574 wait complete. 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.253+0000 7f2a4cfa8640 10 monclient: build_initial_monmap 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.253+0000 7f2a4cfa8640 10 monclient: monmap: 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:42.258 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 1 librados: starting msgr at 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 1 librados: starting objecter 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 1 Processor -- start 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 1 -- start start 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 1 librados: setting wanted keys 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 1 librados: calling monclient init 2024-03-25T14:37:42.259 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 10 monclient: init 2024-03-25T14:37:42.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:42.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:37:42.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a480a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:42.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a4813f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:42.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.254+0000 7f2a4cfa8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a481430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:42.260 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.255+0000 7f2a4cfa8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a48138250 con 0x7f2a481430c0 2024-03-25T14:37:42.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.255+0000 7f2a4cfa8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a48137d50 con 0x7f2a4813f710 2024-03-25T14:37:42.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.255+0000 7f2a4cfa8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a48137ee0 con 0x7f2a480a98f0 2024-03-25T14:37:42.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.255+0000 7f2a4cfa8640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:42.261 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.255+0000 7f2a4cfa8640 10 monclient(hunting): authenticate will time out at 2147.357651s 2024-03-25T14:37:42.267 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.264+0000 7f2a4659c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2a480a98f0 0x7f2a48137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39026/0 (socket says 172.21.15.26:39026) 2024-03-25T14:37:42.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.264+0000 7f2a4659c640 1 -- 172.21.15.26:0/784381106 learned_addr learned my addr 172.21.15.26:0/784381106 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:42.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.265+0000 7f2a36ffd640 10 client.?.objecter ms_handle_connect 0x7f2a480a98f0 2024-03-25T14:37:42.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.265+0000 7f2a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:42.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.265+0000 7f2a36ffd640 10 client.?.objecter ms_handle_connect 0x7f2a4813f710 2024-03-25T14:37:42.268 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.265+0000 7f2a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:42.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.270+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529298618 0 0) 0x7f2a48137d50 con 0x7f2a4813f710 2024-03-25T14:37:42.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.270+0000 7f2a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:42.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.270+0000 7f2a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:42.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.270+0000 7f2a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:42.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.270+0000 7f2a36ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T14:37:42.274 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.270+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a28004eb0 con 0x7f2a4813f710 2024-03-25T14:37:42.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.277+0000 7f2a36ffd640 10 client.?.objecter ms_handle_connect 0x7f2a481430c0 2024-03-25T14:37:42.280 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.277+0000 7f2a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:42.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.303+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012650998 0 0) 0x7f2a48138250 con 0x7f2a481430c0 2024-03-25T14:37:42.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.303+0000 7f2a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:42.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.303+0000 7f2a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:42.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.303+0000 7f2a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:42.306 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.303+0000 7f2a36ffd640 10 monclient(hunting): my global_id is 4710 2024-03-25T14:37:42.307 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.303+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a48137d50 con 0x7f2a481430c0 2024-03-25T14:37:42.362 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.358+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883888420 0 0) 0x7f2a48137ee0 con 0x7f2a480a98f0 2024-03-25T14:37:42.362 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.358+0000 7f2a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:42.362 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.358+0000 7f2a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:42.362 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.358+0000 7f2a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:42.362 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.358+0000 7f2a36ffd640 10 monclient(hunting): my global_id is 4658 2024-03-25T14:37:42.362 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.358+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a48138250 con 0x7f2a480a98f0 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 820388764 0 0) 0x7f2a48137d50 con 0x7f2a481430c0 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a48137ee0 con 0x7f2a481430c0 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a40001b50 con 0x7f2a481430c0 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:42.376 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1557833569 0 0) 0x7f2a28004eb0 con 0x7f2a4813f710 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a48137d50 con 0x7f2a4813f710 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a3c0018f0 con 0x7f2a4813f710 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.372+0000 7f2a36ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.377 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:42.378 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:42.378 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:42.378 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.377+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960169672 0 0) 0x7f2a48138250 con 0x7f2a480a98f0 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.377+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a28004eb0 con 0x7f2a480a98f0 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.377+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a38001b00 con 0x7f2a480a98f0 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.377+0000 7f2a36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.377+0000 7f2a36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.377+0000 7f2a36ffd640 10 monclient(hunting): dump: 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.381 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.382 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:42.382 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.382 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:42.382 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:42.382 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:42.382 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1769887847 0 0) 0x7f2a48137d50 con 0x7f2a4813f710 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 >> v1:172.21.15.26:6790/0 conn(0x7f2a480a98f0 legacy=0x7f2a48137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 >> v1:172.21.15.26:6789/0 conn(0x7f2a481430c0 legacy=0x7f2a481d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 1 monclient: found mon.b 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:42.397 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a480b7730 con 0x7f2a4813f710 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 10 monclient: _finish_auth 0 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 5 monclient: authenticate success, global_id 4696 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 10 monclient: _renew_subs 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a480b7920 con 0x7f2a4813f710 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 10 monclient: _renew_subs 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a481e33d0 con 0x7f2a4813f710 2024-03-25T14:37:42.398 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.393+0000 7f2a4cfa8640 1 librados: init done 2024-03-25T14:37:42.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.395+0000 7f2a4cfa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:42.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.395+0000 7f2a4cfa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:42.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.395+0000 7f2a4cfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:42.399 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.395+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a480627b0 con 0x7f2a4813f710 2024-03-25T14:37:42.675 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a3c0035f0 con 0x7f2a4813f710 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f2a3c003900 con 0x7f2a4813f710 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a36ffd640 10 monclient: dump: 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:42.676 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.672+0000 7f2a44d99640 4 set_mon_vals no callback set 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.673+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f2a3c0198d0 con 0x7f2a4813f710 2024-03-25T14:37:42.677 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.673+0000 7f2a4659c640 10 monclient: get_auth_request con 0x7f2a2806e110 auth_method 0 2024-03-25T14:37:42.678 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.674+0000 7f2a36ffd640 10 client.4696.objecter ms_handle_connect 0x7f2a2806e110 2024-03-25T14:37:42.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.680+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f2a3c0535c0 con 0x7f2a4813f710 2024-03-25T14:37:42.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.680+0000 7f2a36ffd640 10 client.4696.objecter ms_dispatch 0x7f2a48000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:42.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.680+0000 7f2a36ffd640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:42.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.680+0000 7f2a36ffd640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:42.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.681+0000 7f2a36ffd640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T14:37:42.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.681+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a3c04f0d0 con 0x7f2a4813f710 2024-03-25T14:37:42.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.681+0000 7f2a36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:42.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.681+0000 7f2a36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:42.757 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:01.618 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.811 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:01.673 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.841 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:01.703 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.860 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:01.738 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.864 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:01.742 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.882 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:01.760 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.893 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:01.756 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.900 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:01.762 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.917+0000 7f2a4cfa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:42.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.917+0000 7f2a4cfa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:42.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.917+0000 7f2a4cfa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:42.921 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.917+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a481383e0 con 0x7f2a4813f710 2024-03-25T14:37:42.927 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:01.789 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:37:42.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.960+0000 7f2a36ffd640 1 -- 172.21.15.26:0/784381106 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f2a3c051000 con 0x7f2a4813f710 2024-03-25T14:37:42.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.960+0000 7f2a36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:42.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.960+0000 7f2a36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:42.965 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:42.965 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 10 librados: watch_flush enter 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 10 librados: watch_flush exit 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 10 client.4696.objecter successfully canceled tick 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 >> v1:172.21.15.121:6800/288813066 conn(0x7f2a2806e110 legacy=0x7f2a280705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 10 monclient: shutdown 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:42.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 >> v1:172.21.15.121:6789/0 conn(0x7f2a4813f710 legacy=0x7f2a481d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:42.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 shutdown_connections 2024-03-25T14:37:42.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 >> 172.21.15.26:0/784381106 conn(0x7f2a480b7ea0 msgr2=0x7f2a48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:42.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 shutdown_connections 2024-03-25T14:37:42.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 -- 172.21.15.26:0/784381106 wait complete. 2024-03-25T14:37:42.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:42.963+0000 7f2a4cfa8640 1 librados: shutdown 2024-03-25T14:37:42.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:49.785 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:49.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.898+0000 7fdc81ee6640 10 monclient: get_monmap_and_config 2024-03-25T14:37:49.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.898+0000 7fdc81ee6640 10 monclient: build_initial_monmap 2024-03-25T14:37:49.902 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.898+0000 7fdc81ee6640 10 monclient: monmap: 2024-03-25T14:37:49.902 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:49.902 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:49.902 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:49.899534+0000 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:49.899534+0000 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:49.903 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:49.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.899+0000 7fdc81ee6640 1 Processor -- start 2024-03-25T14:37:49.904 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.899+0000 7fdc81ee6640 1 -- start start 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.899+0000 7fdc81ee6640 10 monclient: init 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c0a98f0 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c13f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c1430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c142520 con 0x7fdc7c1430c0 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0b28a0 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.905 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0b2a30 con 0x7fdc7c13f710 2024-03-25T14:37:49.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:49.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.900+0000 7fdc81ee6640 10 monclient(hunting): authenticate will time out at 2155.002939s 2024-03-25T14:37:49.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.903+0000 7fdc7bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdc7c1430c0 0x7fdc7c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36966/0 (socket says 172.21.15.26:36966) 2024-03-25T14:37:49.906 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.903+0000 7fdc7bfff640 1 -- 172.21.15.26:0/1542585709 learned_addr learned my addr 172.21.15.26:0/1542585709 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:49.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.915+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499726987 0 0) 0x7fdc7c142520 con 0x7fdc7c1430c0 2024-03-25T14:37:49.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.916+0000 7fdc7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:37:49.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.916+0000 7fdc7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:49.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.916+0000 7fdc7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:49.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.916+0000 7fdc7a7fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T14:37:49.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.916+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc64003520 con 0x7fdc7c1430c0 2024-03-25T14:37:49.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.923+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68401312 0 0) 0x7fdc7c0b28a0 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.923+0000 7fdc7a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:37:49.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.923+0000 7fdc7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:49.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.923+0000 7fdc7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:49.926 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.923+0000 7fdc7a7fc640 10 monclient(hunting): my global_id is 4699 2024-03-25T14:37:49.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.923+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc7c142520 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.927 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 345921469 0 0) 0x7fdc64003520 con 0x7fdc7c1430c0 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 >> v1:172.21.15.26:6790/0 conn(0x7fdc7c13f710 legacy=0x7fdc7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 >> v1:172.21.15.121:6789/0 conn(0x7fdc7c0a98f0 legacy=0x7fdc7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 1 monclient: found mon.noname-a 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0b44c0 con 0x7fdc7c1430c0 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:49.928 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fdc74001a10 con 0x7fdc7c1430c0 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc7a7fc640 10 monclient: dump: 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:49.929 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:49.930 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:49.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc81ee6640 5 monclient: authenticate success, global_id 4713 2024-03-25T14:37:49.930 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.924+0000 7fdc81ee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:49.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc74001b70 con 0x7fdc7c1430c0 2024-03-25T14:37:49.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:49.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc7a7fc640 1 -- 172.21.15.26:0/1542585709 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fdc74001f80 con 0x7fdc7c1430c0 2024-03-25T14:37:49.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc81ee6640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:49.939 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc81ee6640 4 set_mon_vals no callback set 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc7a7fc640 10 monclient: dump: 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:49.940 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:49.941 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:49.941 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc81ee6640 10 monclient: shutdown 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc81ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.936+0000 7fdc81ee6640 1 -- 172.21.15.26:0/1542585709 >> v1:172.21.15.26:6789/0 conn(0x7fdc7c1430c0 legacy=0x7fdc7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.937+0000 7fdc81ee6640 1 -- 172.21.15.26:0/1542585709 shutdown_connections 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.937+0000 7fdc81ee6640 1 -- 172.21.15.26:0/1542585709 >> 172.21.15.26:0/1542585709 conn(0x7fdc7c0b7ea0 msgr2=0x7fdc7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.937+0000 7fdc81ee6640 1 -- 172.21.15.26:0/1542585709 shutdown_connections 2024-03-25T14:37:49.943 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.937+0000 7fdc81ee6640 1 -- 172.21.15.26:0/1542585709 wait complete. 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.937+0000 7fdc81ee6640 10 monclient: build_initial_monmap 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.937+0000 7fdc81ee6640 10 monclient: monmap: 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:49.944 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 1 librados: starting msgr at 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 1 librados: starting objecter 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 1 Processor -- start 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 1 -- start start 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 1 librados: setting wanted keys 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 1 librados: calling monclient init 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.938+0000 7fdc81ee6640 10 monclient: init 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:49.945 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c0a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c13f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c1430c0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0abb00 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0abde0 con 0x7fdc7c13f710 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c138e20 con 0x7fdc7c1430c0 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:49.946 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.939+0000 7fdc81ee6640 10 monclient(hunting): authenticate will time out at 2155.042121s 2024-03-25T14:37:49.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.942+0000 7fdc7b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdc7c0a98f0 0x7fdc7c1385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:36996/0 (socket says 172.21.15.26:36996) 2024-03-25T14:37:49.947 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.942+0000 7fdc7b7fe640 1 -- 172.21.15.26:0/276524672 learned_addr learned my addr 172.21.15.26:0/276524672 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:49.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.945+0000 7fdc63fff640 10 client.?.objecter ms_handle_connect 0x7fdc7c0a98f0 2024-03-25T14:37:49.948 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.945+0000 7fdc63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:49.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.950+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344413664 0 0) 0x7fdc7c0abb00 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.950+0000 7fdc63fff640 10 monclient(hunting): _negotiate 2024-03-25T14:37:49.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.950+0000 7fdc63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:49.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.950+0000 7fdc63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:49.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.950+0000 7fdc63fff640 10 monclient(hunting): my global_id is 4716 2024-03-25T14:37:49.954 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.950+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc68004eb0 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.953+0000 7fdc63fff640 10 client.?.objecter ms_handle_connect 0x7fdc7c1430c0 2024-03-25T14:37:49.956 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.953+0000 7fdc63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:49.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2287283068 0 0) 0x7fdc68004eb0 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.963 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc7c0abb00 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 10 client.?.objecter ms_handle_connect 0x7fdc7c13f710 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fdc6c001a10 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 10 monclient(hunting): dump: 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:49.964 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:49.965 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:49.965 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:49.965 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:49.965 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.960+0000 7fdc63fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1064192702 0 0) 0x7fdc7c0abb00 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 >> v1:172.21.15.26:6790/0 conn(0x7fdc7c1430c0 legacy=0x7fdc7c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 >> v1:172.21.15.121:6789/0 conn(0x7fdc7c13f710 legacy=0x7fdc7c136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 10 monclient: _finish_hunting 0 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 1 monclient: found mon.a 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c137b00 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 10 monclient: _finish_auth 0 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 10 monclient: _check_auth_tickets 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 5 monclient: authenticate success, global_id 4716 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 10 monclient: _renew_subs 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c1e34b0 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 10 client.4716.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 10 monclient: _renew_subs 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c1e3820 con 0x7fdc7c0a98f0 2024-03-25T14:37:49.971 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.966+0000 7fdc81ee6640 1 librados: init done 2024-03-25T14:37:49.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.967+0000 7fdc81ee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:49.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.967+0000 7fdc81ee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:49.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.967+0000 7fdc81ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:37:49.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:49.967+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c138fb0 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc6c001cd0 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:50.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fdc6c003a60 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:50.287 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc63fff640 10 monclient: dump: 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:50.288 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.283+0000 7fdc79ffb640 4 set_mon_vals no callback set 2024-03-25T14:37:50.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.284+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fdc6c003c40 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.285+0000 7fdc7affd640 10 monclient: get_auth_request con 0x7fdc6806dc90 auth_method 0 2024-03-25T14:37:50.289 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.285+0000 7fdc63fff640 10 client.4716.objecter ms_handle_connect 0x7fdc6806dc90 2024-03-25T14:37:50.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7fdc6c054980 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 10 client.4716.objecter ms_dispatch 0x7fdc7c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:50.296 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 3 client.4716.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:50.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 3 client.4716.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:50.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 20 client.4716.objecter dump_active .. 0 homeless 2024-03-25T14:37:50.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc6c0502c0 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:50.297 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.293+0000 7fdc63fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:50.533 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.530+0000 7fdc81ee6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:50.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.530+0000 7fdc81ee6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:50.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.530+0000 7fdc81ee6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-03-25T14:37:50.534 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.530+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc7c0627b0 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.556+0000 7fdc63fff640 1 -- 172.21.15.26:0/276524672 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7fdc6c0523c0 con 0x7fdc7c0a98f0 2024-03-25T14:37:50.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.556+0000 7fdc63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:50.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.556+0000 7fdc63fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:50.560 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:50.560 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:50.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 10 librados: watch_flush enter 2024-03-25T14:37:50.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 10 librados: watch_flush exit 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 20 client.4716.objecter shutdown clearing up homeless session... 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 10 client.4716.objecter successfully canceled tick 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 >> v1:172.21.15.121:6800/288813066 conn(0x7fdc6806dc90 legacy=0x7fdc68070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 10 monclient: shutdown 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 >> v1:172.21.15.26:6789/0 conn(0x7fdc7c0a98f0 legacy=0x7fdc7c1385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 shutdown_connections 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.558+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 >> 172.21.15.26:0/276524672 conn(0x7fdc7c0b7ea0 msgr2=0x7fdc7c14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.559+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 shutdown_connections 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.559+0000 7fdc81ee6640 1 -- 172.21.15.26:0/276524672 wait complete. 2024-03-25T14:37:50.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:50.559+0000 7fdc81ee6640 1 librados: shutdown 2024-03-25T14:37:50.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:37:57.381 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.494+0000 7fabea77a640 10 monclient: get_monmap_and_config 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.494+0000 7fabea77a640 10 monclient: build_initial_monmap 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.494+0000 7fabea77a640 10 monclient: monmap: 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:37:57.495717+0000 2024-03-25T14:37:57.498 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:37:57.495717+0000 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.496+0000 7fabea77a640 1 Processor -- start 2024-03-25T14:37:57.499 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.496+0000 7fabea77a640 1 -- start start 2024-03-25T14:37:57.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.496+0000 7fabea77a640 10 monclient: init 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient(hunting): picked mon.noname-a con 0x7fabe40b1ef0 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient(hunting): picked mon.noname-b con 0x7fabe41473a0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient(hunting): picked mon.noname-c con 0x7fabe4150070 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe40b2570 con 0x7fabe40b1ef0 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe4146c20 con 0x7fabe4150070 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe4146e80 con 0x7fabe41473a0 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:57.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.497+0000 7fabea77a640 10 monclient(hunting): authenticate will time out at 2162.599969s 2024-03-25T14:37:57.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.504+0000 7fabe8d17640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fabe4150070 0x7fabe4152460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:56558/0 (socket says 172.21.15.26:56558) 2024-03-25T14:37:57.507 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.504+0000 7fabe8d17640 1 -- 172.21.15.26:0/3206791868 learned_addr learned my addr 172.21.15.26:0/3206791868 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:57.527 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.524+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096176431 0 0) 0x7fabe40b2570 con 0x7fabe40b1ef0 2024-03-25T14:37:57.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.524+0000 7fabe2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:57.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.524+0000 7fabe2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:57.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.524+0000 7fabe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:57.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.524+0000 7fabe2ffd640 10 monclient(hunting): my global_id is 4719 2024-03-25T14:37:57.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.524+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc003520 con 0x7fabe40b1ef0 2024-03-25T14:37:57.529 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.526+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338822180 0 0) 0x7fabe4146c20 con 0x7fabe4150070 2024-03-25T14:37:57.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.526+0000 7fabe2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:57.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.526+0000 7fabe2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:57.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.526+0000 7fabe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:57.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.526+0000 7fabe2ffd640 10 monclient(hunting): my global_id is 4705 2024-03-25T14:37:57.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.526+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe40b2570 con 0x7fabe4150070 2024-03-25T14:37:57.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.528+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413971141 0 0) 0x7fabe4146e80 con 0x7fabe41473a0 2024-03-25T14:37:57.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.528+0000 7fabe2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:37:57.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.528+0000 7fabe2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:57.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.528+0000 7fabe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:57.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.528+0000 7fabe2ffd640 10 monclient(hunting): my global_id is 4667 2024-03-25T14:37:57.532 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.528+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4146c20 con 0x7fabe41473a0 2024-03-25T14:37:57.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2114950450 0 0) 0x7fabe4146c20 con 0x7fabe41473a0 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 >> v1:172.21.15.121:6789/0 conn(0x7fabe4150070 legacy=0x7fabe4152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 >> v1:172.21.15.26:6789/0 conn(0x7fabe40b1ef0 legacy=0x7fabe40b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 1 monclient: found mon.noname-b 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe406e1b0 con 0x7fabe41473a0 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: _finish_auth 0 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fabd80015b0 con 0x7fabe41473a0 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:57.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabe2ffd640 10 monclient: dump: 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabea77a640 5 monclient: authenticate success, global_id 4667 2024-03-25T14:37:57.548 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.543+0000 7fabea77a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:37:57.555 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd8001790 con 0x7fabe41473a0 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabe2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabe2ffd640 1 -- 172.21.15.26:0/3206791868 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fabd8001be0 con 0x7fabe41473a0 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabe2ffd640 10 monclient: dump: 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:57.556 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:57.557 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:57.557 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:57.557 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:57.557 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:57.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabea77a640 10 monclient: get_monmap_and_config success 2024-03-25T14:37:57.557 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.552+0000 7fabea77a640 4 set_mon_vals no callback set 2024-03-25T14:37:57.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.553+0000 7fabea77a640 10 monclient: shutdown 2024-03-25T14:37:57.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.553+0000 7fabea77a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:57.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.553+0000 7fabea77a640 1 -- 172.21.15.26:0/3206791868 >> v1:172.21.15.26:6790/0 conn(0x7fabe41473a0 legacy=0x7fabe4143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:57.559 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.553+0000 7fabea77a640 1 -- 172.21.15.26:0/3206791868 shutdown_connections 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.553+0000 7fabea77a640 1 -- 172.21.15.26:0/3206791868 >> 172.21.15.26:0/3206791868 conn(0x7fabe40b7ea0 msgr2=0x7fabe40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.553+0000 7fabea77a640 1 -- 172.21.15.26:0/3206791868 shutdown_connections 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.554+0000 7fabea77a640 1 -- 172.21.15.26:0/3206791868 wait complete. 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.554+0000 7fabea77a640 10 monclient: build_initial_monmap 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.554+0000 7fabea77a640 10 monclient: monmap: 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:37:57.560 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 1 librados: starting msgr at 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 1 librados: starting objecter 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 1 Processor -- start 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 1 -- start start 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 1 librados: setting wanted keys 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 1 librados: calling monclient init 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.555+0000 7fabea77a640 10 monclient: init 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient: _reopen_session rank -1 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient(hunting): picked mon.noname-b con 0x7fabe40b1ef0 addr v1:172.21.15.121:6789/0 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient(hunting): picked mon.noname-c con 0x7fabe41473a0 addr v1:172.21.15.26:6790/0 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient(hunting): picked mon.noname-a con 0x7fabe4150070 addr v1:172.21.15.26:6789/0 2024-03-25T14:37:57.561 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe41dec00 con 0x7fabe4150070 2024-03-25T14:37:57.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe406f810 con 0x7fabe40b1ef0 2024-03-25T14:37:57.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe4140ae0 con 0x7fabe41473a0 2024-03-25T14:37:57.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient(hunting): _renew_subs 2024-03-25T14:37:57.562 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.556+0000 7fabea77a640 10 monclient(hunting): authenticate will time out at 2162.658766s 2024-03-25T14:37:57.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.573+0000 7fabe8d17640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fabe4150070 0x7fabe41e6fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:41626/0 (socket says 172.21.15.26:41626) 2024-03-25T14:37:57.577 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.573+0000 7fabe8d17640 1 -- 172.21.15.26:0/1213948020 learned_addr learned my addr 172.21.15.26:0/1213948020 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:37:57.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.574+0000 7fabe0ff9640 10 client.?.objecter ms_handle_connect 0x7fabe4150070 2024-03-25T14:37:57.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.574+0000 7fabe0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:57.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.575+0000 7fabe0ff9640 10 client.?.objecter ms_handle_connect 0x7fabe40b1ef0 2024-03-25T14:37:57.578 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.575+0000 7fabe0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:57.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.581+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560554296 0 0) 0x7fabe406f810 con 0x7fabe40b1ef0 2024-03-25T14:37:57.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.581+0000 7fabe0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:37:57.584 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.581+0000 7fabe0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:57.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.581+0000 7fabe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:57.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.581+0000 7fabe0ff9640 10 monclient(hunting): my global_id is 4708 2024-03-25T14:37:57.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.581+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd0004eb0 con 0x7fabe40b1ef0 2024-03-25T14:37:57.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 10 client.?.objecter ms_handle_connect 0x7fabe41473a0 2024-03-25T14:37:57.585 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:37:57.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449030563 0 0) 0x7fabe41dec00 con 0x7fabe4150070 2024-03-25T14:37:57.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:37:57.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:57.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:57.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 10 monclient(hunting): my global_id is 4722 2024-03-25T14:37:57.586 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.582+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe406f810 con 0x7fabe4150070 2024-03-25T14:37:57.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.598+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897140057 0 0) 0x7fabe4140ae0 con 0x7fabe41473a0 2024-03-25T14:37:57.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.598+0000 7fabe0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:37:57.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.598+0000 7fabe0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:37:57.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.598+0000 7fabe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:37:57.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.598+0000 7fabe0ff9640 10 monclient(hunting): my global_id is 4670 2024-03-25T14:37:57.601 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.598+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe41dec00 con 0x7fabe41473a0 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3909361756 0 0) 0x7fabd0004eb0 con 0x7fabe40b1ef0 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe4140ae0 con 0x7fabe40b1ef0 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fabd4001950 con 0x7fabe40b1ef0 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 10 monclient(hunting): dump: 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:57.607 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:57.608 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.603+0000 7fabe0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.612+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2558234743 0 0) 0x7fabe4140ae0 con 0x7fabe40b1ef0 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 >> v1:172.21.15.26:6790/0 conn(0x7fabe41473a0 legacy=0x7fabe4138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 >> v1:172.21.15.26:6789/0 conn(0x7fabe4150070 legacy=0x7fabe41e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 1 monclient: found mon.b 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:37:57.617 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe413e480 con 0x7fabe40b1ef0 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 10 monclient: _finish_auth 0 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabe0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 5 monclient: authenticate success, global_id 4708 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 10 monclient: _renew_subs 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe413e730 con 0x7fabe40b1ef0 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 10 monclient: _renew_subs 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:57.618 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe41410e0 con 0x7fabe40b1ef0 2024-03-25T14:37:57.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.613+0000 7fabea77a640 1 librados: init done 2024-03-25T14:37:57.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.614+0000 7fabea77a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:57.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.614+0000 7fabea77a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:57.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.614+0000 7fabea77a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:57.619 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.614+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe413e970 con 0x7fabe40b1ef0 2024-03-25T14:37:57.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd4003610 con 0x7fabe40b1ef0 2024-03-25T14:37:57.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:37:57.911 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fabd4003780 con 0x7fabe40b1ef0 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 10 monclient: dump: 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:37:57.912 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe27fc640 4 set_mon_vals no callback set 2024-03-25T14:37:57.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.908+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fabd4019750 con 0x7fabe40b1ef0 2024-03-25T14:37:57.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.909+0000 7fabe37fe640 10 monclient: get_auth_request con 0x7fabd006ddf0 auth_method 0 2024-03-25T14:37:57.913 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.909+0000 7fabe0ff9640 10 client.4708.objecter ms_handle_connect 0x7fabd006ddf0 2024-03-25T14:37:57.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7fabd40533e0 con 0x7fabe40b1ef0 2024-03-25T14:37:57.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 10 client.4708.objecter ms_dispatch 0x7fabe4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:37:57.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:37:57.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:37:57.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T14:37:57.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabd404ed20 con 0x7fabe40b1ef0 2024-03-25T14:37:57.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:37:57.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:57.914+0000 7fabe0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:37:58.150 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.147+0000 7fabea77a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:58.150 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.147+0000 7fabea77a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:58.150 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.147+0000 7fabea77a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:37:58.151 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.147+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabe40627b0 con 0x7fabe40b1ef0 2024-03-25T14:37:58.193 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.190+0000 7fabe0ff9640 1 -- 172.21.15.26:0/1213948020 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7fabd4050e20 con 0x7fabe40b1ef0 2024-03-25T14:37:58.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.190+0000 7fabe0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:37:58.194 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.190+0000 7fabe0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:37:58.194 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:37:58.194 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 10 librados: watch_flush enter 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 10 librados: watch_flush exit 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 10 client.4708.objecter successfully canceled tick 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 >> v1:172.21.15.121:6800/288813066 conn(0x7fabd006ddf0 legacy=0x7fabd00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 10 monclient: shutdown 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 >> v1:172.21.15.121:6789/0 conn(0x7fabe40b1ef0 legacy=0x7fabe41df090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 shutdown_connections 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 >> 172.21.15.26:0/1213948020 conn(0x7fabe40b7ea0 msgr2=0x7fabe413a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 shutdown_connections 2024-03-25T14:37:58.197 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 -- 172.21.15.26:0/1213948020 wait complete. 2024-03-25T14:37:58.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:37:58.193+0000 7fabea77a640 1 librados: shutdown 2024-03-25T14:37:58.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:38:02.840 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:21.701 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.848 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:21.710 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.884 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:21.746 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.896 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:21.774 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.901 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:21.780 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.912 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:21.774 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.917 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:21.779 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.940 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:21.818 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:02.941 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:21.803 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:05.015 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:38:05.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.129+0000 7ffb2013a640 10 monclient: get_monmap_and_config 2024-03-25T14:38:05.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.129+0000 7ffb2013a640 10 monclient: build_initial_monmap 2024-03-25T14:38:05.132 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.129+0000 7ffb2013a640 10 monclient: monmap: 2024-03-25T14:38:05.132 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:05.132 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:38:05.129971+0000 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:38:05.129971+0000 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:38:05.133 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.129+0000 7ffb2013a640 1 Processor -- start 2024-03-25T14:38:05.134 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 1 -- start start 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient: init 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient(hunting): picked mon.noname-c con 0x7ffb180b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient(hunting): picked mon.noname-b con 0x7ffb180b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1814be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180af940 con 0x7ffb1814be90 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180b0a70 con 0x7ffb180b4d60 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180b0d50 con 0x7ffb180b1730 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:05.135 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.130+0000 7ffb2013a640 10 monclient(hunting): authenticate will time out at 2170.233583s 2024-03-25T14:38:05.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.137+0000 7ffb1ded6640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ffb180b4d60 0x7ffb1813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42836/0 (socket says 172.21.15.26:42836) 2024-03-25T14:38:05.140 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.137+0000 7ffb1ded6640 1 -- 172.21.15.26:0/724689065 learned_addr learned my addr 172.21.15.26:0/724689065 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:05.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.154+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432574899 0 0) 0x7ffb180b0a70 con 0x7ffb180b4d60 2024-03-25T14:38:05.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.154+0000 7ffb1ced4640 10 monclient(hunting): _negotiate 2024-03-25T14:38:05.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.154+0000 7ffb1ced4640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:05.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.154+0000 7ffb1ced4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:05.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.154+0000 7ffb1ced4640 10 monclient(hunting): my global_id is 4711 2024-03-25T14:38:05.157 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.154+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003520 con 0x7ffb180b4d60 2024-03-25T14:38:05.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.166+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568434108 0 0) 0x7ffb180af940 con 0x7ffb1814be90 2024-03-25T14:38:05.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.166+0000 7ffb1ced4640 10 monclient(hunting): _negotiate 2024-03-25T14:38:05.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.166+0000 7ffb1ced4640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:05.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.166+0000 7ffb1ced4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:05.169 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.166+0000 7ffb1ced4640 10 monclient(hunting): my global_id is 4725 2024-03-25T14:38:05.170 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.166+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb180b0a70 con 0x7ffb1814be90 2024-03-25T14:38:05.171 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 820848514 0 0) 0x7ffb04003520 con 0x7ffb180b4d60 2024-03-25T14:38:05.171 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 >> v1:172.21.15.26:6790/0 conn(0x7ffb180b1730 legacy=0x7ffb180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.171 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 >> v1:172.21.15.26:6789/0 conn(0x7ffb1814be90 legacy=0x7ffb1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: _finish_hunting 0 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 1 monclient: found mon.noname-c 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180b1030 con 0x7ffb180b4d60 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: _finish_auth 0 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: _check_auth_tickets 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ffb14001690 con 0x7ffb180b4d60 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb1ced4640 10 monclient: dump: 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.172 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb2013a640 5 monclient: authenticate success, global_id 4711 2024-03-25T14:38:05.173 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.168+0000 7ffb2013a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb140017f0 con 0x7ffb180b4d60 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb1ced4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb1ced4640 1 -- 172.21.15.26:0/724689065 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ffb14001c00 con 0x7ffb180b4d60 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb2013a640 10 monclient: get_monmap_and_config success 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb2013a640 4 set_mon_vals no callback set 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb1ced4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb1ced4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.182+0000 7ffb1ced4640 10 monclient: dump: 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.186 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:05.187 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:05.187 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:05.187 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:05.187 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:05.187 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.189 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 10 monclient: shutdown 2024-03-25T14:38:05.189 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:05.189 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 1 -- 172.21.15.26:0/724689065 >> v1:172.21.15.121:6789/0 conn(0x7ffb180b4d60 legacy=0x7ffb1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.189 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 1 -- 172.21.15.26:0/724689065 shutdown_connections 2024-03-25T14:38:05.189 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 1 -- 172.21.15.26:0/724689065 >> 172.21.15.26:0/724689065 conn(0x7ffb180617a0 msgr2=0x7ffb18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 1 -- 172.21.15.26:0/724689065 shutdown_connections 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.183+0000 7ffb2013a640 1 -- 172.21.15.26:0/724689065 wait complete. 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.184+0000 7ffb2013a640 10 monclient: build_initial_monmap 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.184+0000 7ffb2013a640 10 monclient: monmap: 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:38:05.190 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.184+0000 7ffb2013a640 1 librados: starting msgr at 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.184+0000 7ffb2013a640 1 librados: starting objecter 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 Processor -- start 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 -- start start 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 librados: setting wanted keys 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 librados: calling monclient init 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient: init 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:38:05.191 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient(hunting): picked mon.noname-b con 0x7ffb180b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb180b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1814be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18141ce0 con 0x7ffb180b4d60 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb18142610 con 0x7ffb180b1730 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb181dae80 con 0x7ffb1814be90 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:05.192 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.185+0000 7ffb2013a640 10 monclient(hunting): authenticate will time out at 2170.288386s 2024-03-25T14:38:05.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.199+0000 7ffb1ded6640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ffb180b1730 0x7ffb18142b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:42908/0 (socket says 172.21.15.26:42908) 2024-03-25T14:38:05.202 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.199+0000 7ffb1ded6640 1 -- 172.21.15.26:0/3035440800 learned_addr learned my addr 172.21.15.26:0/3035440800 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:05.203 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.200+0000 7ffb0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb180b1730 2024-03-25T14:38:05.205 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.200+0000 7ffb0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:05.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.204+0000 7ffb0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb1814be90 2024-03-25T14:38:05.207 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.204+0000 7ffb0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:05.216 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.213+0000 7ffb0e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb180b4d60 2024-03-25T14:38:05.217 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.213+0000 7ffb0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:05.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.216+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364781144 0 0) 0x7ffb18142610 con 0x7ffb180b1730 2024-03-25T14:38:05.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.216+0000 7ffb0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:38:05.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.216+0000 7ffb0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:05.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.216+0000 7ffb0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:05.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.216+0000 7ffb0e7fc640 10 monclient(hunting): my global_id is 4714 2024-03-25T14:38:05.220 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc004eb0 con 0x7ffb180b1730 2024-03-25T14:38:05.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118716149 0 0) 0x7ffb181dae80 con 0x7ffb1814be90 2024-03-25T14:38:05.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:38:05.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:05.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:05.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 10 monclient(hunting): my global_id is 4676 2024-03-25T14:38:05.221 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.217+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18142610 con 0x7ffb1814be90 2024-03-25T14:38:05.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.228+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482265916 0 0) 0x7ffb18141ce0 con 0x7ffb180b4d60 2024-03-25T14:38:05.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.228+0000 7ffb0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:38:05.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.228+0000 7ffb0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:05.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.228+0000 7ffb0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:05.231 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.228+0000 7ffb0e7fc640 10 monclient(hunting): my global_id is 4728 2024-03-25T14:38:05.232 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.228+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb181dae80 con 0x7ffb180b4d60 2024-03-25T14:38:05.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.235+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1606886910 0 0) 0x7ffb18142610 con 0x7ffb1814be90 2024-03-25T14:38:05.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.235+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb18141ce0 con 0x7ffb1814be90 2024-03-25T14:38:05.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.235+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ffb10001b90 con 0x7ffb1814be90 2024-03-25T14:38:05.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.235+0000 7ffb0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:05.239 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.235+0000 7ffb0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.235+0000 7ffb0e7fc640 10 monclient(hunting): dump: 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.240 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.236+0000 7ffb0e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.238+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703704802 0 0) 0x7ffafc004eb0 con 0x7ffb180b1730 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.238+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb18142610 con 0x7ffb180b1730 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.238+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ffb14001c00 con 0x7ffb180b1730 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.238+0000 7ffb0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.238+0000 7ffb0e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.238+0000 7ffb0e7fc640 10 monclient(hunting): dump: 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.241 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.242 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:05.242 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:05.242 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:05.242 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:05.242 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:05.242 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003361383 0 0) 0x7ffb18141ce0 con 0x7ffb1814be90 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 >> v1:172.21.15.121:6789/0 conn(0x7ffb180b1730 legacy=0x7ffb18142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 >> v1:172.21.15.26:6789/0 conn(0x7ffb180b4d60 legacy=0x7ffb181db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 1 monclient: found mon.c 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb181db010 con 0x7ffb1814be90 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 5 monclient: authenticate success, global_id 4676 2024-03-25T14:38:05.255 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 10 monclient: _renew_subs 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18143240 con 0x7ffb1814be90 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 10 monclient: _renew_subs 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1813bd20 con 0x7ffb1814be90 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.251+0000 7ffb2013a640 1 librados: init done 2024-03-25T14:38:05.256 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.252+0000 7ffb2013a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:05.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.252+0000 7ffb2013a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:05.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.252+0000 7ffb2013a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:38:05.257 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.252+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb181427a0 con 0x7ffb1814be90 2024-03-25T14:38:05.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb10002040 con 0x7ffb1814be90 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7ffb10003b30 con 0x7ffb1814be90 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0e7fc640 10 monclient: dump: 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:05.564 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:05.565 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:05.565 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:05.565 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:05.565 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:05.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.560+0000 7ffb0ffff640 4 set_mon_vals no callback set 2024-03-25T14:38:05.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.561+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7ffb10019b00 con 0x7ffb1814be90 2024-03-25T14:38:05.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.562+0000 7ffb1ded6640 10 monclient: get_auth_request con 0x7ffafc06df20 auth_method 0 2024-03-25T14:38:05.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.562+0000 7ffb0e7fc640 10 client.4676.objecter ms_handle_connect 0x7ffafc06df20 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7ffb10053730 con 0x7ffb1814be90 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 10 client.4676.objecter ms_dispatch 0x7ffb18000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb1004f240 con 0x7ffb1814be90 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:05.570 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.566+0000 7ffb0e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:38:05.804 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.801+0000 7ffb2013a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:05.805 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.801+0000 7ffb2013a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:05.805 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.801+0000 7ffb2013a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6790/0 2024-03-25T14:38:05.805 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.801+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb180627b0 con 0x7ffb1814be90 2024-03-25T14:38:05.885 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.882+0000 7ffb0e7fc640 1 -- 172.21.15.26:0/3035440800 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7ffb10051170 con 0x7ffb1814be90 2024-03-25T14:38:05.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.882+0000 7ffb0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:05.886 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.882+0000 7ffb0e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:38:05.886 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:38:05.886 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:38:05.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 10 librados: watch_flush enter 2024-03-25T14:38:05.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 10 librados: watch_flush exit 2024-03-25T14:38:05.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 10 client.4676.objecter successfully canceled tick 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 >> v1:172.21.15.121:6800/288813066 conn(0x7ffafc06df20 legacy=0x7ffafc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 10 monclient: shutdown 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.884+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 >> v1:172.21.15.26:6790/0 conn(0x7ffb1814be90 legacy=0x7ffb18140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.885+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 shutdown_connections 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.885+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 >> 172.21.15.26:0/3035440800 conn(0x7ffb180617a0 msgr2=0x7ffb180b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.885+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 shutdown_connections 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.885+0000 7ffb2013a640 1 -- 172.21.15.26:0/3035440800 wait complete. 2024-03-25T14:38:05.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:05.885+0000 7ffb2013a640 1 librados: shutdown 2024-03-25T14:38:05.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:38:12.709 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:38:12.823 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.819+0000 7f467919d640 10 monclient: get_monmap_and_config 2024-03-25T14:38:12.823 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.819+0000 7f467919d640 10 monclient: build_initial_monmap 2024-03-25T14:38:12.823 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.819+0000 7f467919d640 10 monclient: monmap: 2024-03-25T14:38:12.823 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:12.823 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:12.823 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:38:12.820714+0000 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:38:12.820714+0000 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:38:12.824 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.820+0000 7f467919d640 1 Processor -- start 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.820+0000 7f467919d640 1 -- start start 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.820+0000 7f467919d640 10 monclient: init 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient(hunting): picked mon.noname-a con 0x7f46740a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient(hunting): picked mon.noname-b con 0x7f467413f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient(hunting): picked mon.noname-c con 0x7f46741430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674142520 con 0x7f46740a98f0 2024-03-25T14:38:12.825 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b28a0 con 0x7f46741430c0 2024-03-25T14:38:12.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b2a30 con 0x7f467413f710 2024-03-25T14:38:12.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:12.826 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.821+0000 7f467919d640 10 monclient(hunting): authenticate will time out at 2177.924184s 2024-03-25T14:38:12.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.826+0000 7f467359e640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f46741430c0 0x7f467414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:43148/0 (socket says 172.21.15.26:43148) 2024-03-25T14:38:12.830 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.826+0000 7f467359e640 1 -- 172.21.15.26:0/3054171664 learned_addr learned my addr 172.21.15.26:0/3054171664 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:12.835 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.832+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080963968 0 0) 0x7f46740b28a0 con 0x7f46741430c0 2024-03-25T14:38:12.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.832+0000 7f4671d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:38:12.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.832+0000 7f4671d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:12.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.832+0000 7f4671d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:12.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.832+0000 7f4671d9b640 10 monclient(hunting): my global_id is 4717 2024-03-25T14:38:12.836 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.832+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c003520 con 0x7f46741430c0 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325473361 0 0) 0x7f465c003520 con 0x7f46741430c0 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 >> v1:172.21.15.26:6790/0 conn(0x7f467413f710 legacy=0x7f46741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 >> v1:172.21.15.26:6789/0 conn(0x7f46740a98f0 legacy=0x7f46740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: _finish_hunting 0 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 1 monclient: found mon.noname-c 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740b44c0 con 0x7f46741430c0 2024-03-25T14:38:12.841 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: _finish_auth 0 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: _check_auth_tickets 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f466c001a10 con 0x7f46741430c0 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f4671d9b640 10 monclient: dump: 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:12.842 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:12.843 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:12.843 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:12.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f467919d640 5 monclient: authenticate success, global_id 4717 2024-03-25T14:38:12.843 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.837+0000 7f467919d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:12.850 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f466c001be0 con 0x7f46741430c0 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f4671d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f4671d9b640 1 -- 172.21.15.26:0/3054171664 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f466c002030 con 0x7f46741430c0 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f467919d640 10 monclient: get_monmap_and_config success 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f467919d640 4 set_mon_vals no callback set 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f4671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f4671d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.847+0000 7f4671d9b640 10 monclient: dump: 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:12.851 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:12.852 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:12.852 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:12.852 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:12.852 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:12.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 10 monclient: shutdown 2024-03-25T14:38:12.853 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 1 -- 172.21.15.26:0/3054171664 >> v1:172.21.15.121:6789/0 conn(0x7f46741430c0 legacy=0x7f467414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 1 -- 172.21.15.26:0/3054171664 shutdown_connections 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 1 -- 172.21.15.26:0/3054171664 >> 172.21.15.26:0/3054171664 conn(0x7f46740b7ea0 msgr2=0x7f46740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 1 -- 172.21.15.26:0/3054171664 shutdown_connections 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.848+0000 7f467919d640 1 -- 172.21.15.26:0/3054171664 wait complete. 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.849+0000 7f467919d640 10 monclient: build_initial_monmap 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.849+0000 7f467919d640 10 monclient: monmap: 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:38:12.854 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 1 librados: starting msgr at 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 1 librados: starting objecter 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 1 Processor -- start 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 1 -- start start 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 1 librados: setting wanted keys 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 1 librados: calling monclient init 2024-03-25T14:38:12.855 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.850+0000 7f467919d640 10 monclient: init 2024-03-25T14:38:12.856 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:12.856 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:38:12.856 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient(hunting): picked mon.noname-c con 0x7f46740a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:12.856 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient(hunting): picked mon.noname-b con 0x7f467413f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:12.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient(hunting): picked mon.noname-a con 0x7f46741430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:12.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674137640 con 0x7f46741430c0 2024-03-25T14:38:12.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b1730 con 0x7f467413f710 2024-03-25T14:38:12.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46741399c0 con 0x7f46740a98f0 2024-03-25T14:38:12.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:12.857 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.851+0000 7f467919d640 10 monclient(hunting): authenticate will time out at 2177.953871s 2024-03-25T14:38:12.867 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.864+0000 7f4672d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f46740a98f0 0x7f46741d4450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48252/0 (socket says 172.21.15.26:48252) 2024-03-25T14:38:12.867 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.864+0000 7f4672d9d640 1 -- 172.21.15.26:0/1416496416 learned_addr learned my addr 172.21.15.26:0/1416496416 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:12.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.865+0000 7f465b7fe640 10 client.?.objecter ms_handle_connect 0x7f46740a98f0 2024-03-25T14:38:12.868 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.865+0000 7f465b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:12.873 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.870+0000 7f465b7fe640 10 client.?.objecter ms_handle_connect 0x7f467413f710 2024-03-25T14:38:12.874 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.870+0000 7f465b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:12.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.875+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821773182 0 0) 0x7f46741399c0 con 0x7f46740a98f0 2024-03-25T14:38:12.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.875+0000 7f465b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:38:12.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.875+0000 7f465b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:12.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.875+0000 7f465b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:12.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.875+0000 7f465b7fe640 10 monclient(hunting): my global_id is 4679 2024-03-25T14:38:12.878 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.875+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660004eb0 con 0x7f46740a98f0 2024-03-25T14:38:12.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.879+0000 7f465b7fe640 10 client.?.objecter ms_handle_connect 0x7f46741430c0 2024-03-25T14:38:12.882 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.879+0000 7f465b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:12.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.885+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104970134 0 0) 0x7f46740b1730 con 0x7f467413f710 2024-03-25T14:38:12.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.885+0000 7f465b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:38:12.888 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.885+0000 7f465b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:12.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.885+0000 7f465b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:12.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.885+0000 7f465b7fe640 10 monclient(hunting): my global_id is 4720 2024-03-25T14:38:12.889 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.885+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46741399c0 con 0x7f467413f710 2024-03-25T14:38:12.898 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3533817605 0 0) 0x7f4660004eb0 con 0x7f46740a98f0 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46740b1730 con 0x7f46740a98f0 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4664001a30 con 0x7f46740a98f0 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 10 monclient(hunting): dump: 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:12.899 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:12.900 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:12.900 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:12.900 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.895+0000 7f465b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.905+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 436585523 0 0) 0x7f46741399c0 con 0x7f467413f710 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.905+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4660004eb0 con 0x7f467413f710 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.905+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4668001810 con 0x7f467413f710 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.905+0000 7f465b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.905+0000 7f465b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.905+0000 7f465b7fe640 10 monclient(hunting): dump: 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.908 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.906+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420382813 0 0) 0x7f4674137640 con 0x7f46741430c0 2024-03-25T14:38:12.909 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.906+0000 7f465b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:38:12.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.906+0000 7f465b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:12.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.906+0000 7f465b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:12.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.906+0000 7f465b7fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T14:38:12.910 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.906+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46741399c0 con 0x7f46741430c0 2024-03-25T14:38:12.917 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3544090579 0 0) 0x7f4660004eb0 con 0x7f467413f710 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 >> v1:172.21.15.26:6790/0 conn(0x7f46740a98f0 legacy=0x7f46741d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 >> v1:172.21.15.26:6789/0 conn(0x7f46741430c0 legacy=0x7f46741e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 1 monclient: found mon.b 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46741d4b60 con 0x7f467413f710 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f465b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 5 monclient: authenticate success, global_id 4720 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 10 monclient: _renew_subs 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:12.918 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46741d4e10 con 0x7f467413f710 2024-03-25T14:38:12.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:38:12.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 10 monclient: _renew_subs 2024-03-25T14:38:12.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:12.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46741db640 con 0x7f467413f710 2024-03-25T14:38:12.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.914+0000 7f467919d640 1 librados: init done 2024-03-25T14:38:12.919 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.915+0000 7f467919d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:12.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.915+0000 7f467919d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:12.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.915+0000 7f467919d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:12.920 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:12.915+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46740b3ef0 con 0x7f467413f710 2024-03-25T14:38:13.198 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4668001d00 con 0x7f467413f710 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f465b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4668003790 con 0x7f467413f710 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f465b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f465b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f465b7fe640 10 monclient: dump: 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:13.199 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.192+0000 7f467159a640 4 set_mon_vals no callback set 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.193+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f46680197a0 con 0x7f467413f710 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.194+0000 7f4672d9d640 10 monclient: get_auth_request con 0x7f466006df10 auth_method 0 2024-03-25T14:38:13.200 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.194+0000 7f465b7fe640 10 client.4720.objecter ms_handle_connect 0x7f466006df10 2024-03-25T14:38:13.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.198+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f4668053500 con 0x7f467413f710 2024-03-25T14:38:13.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.198+0000 7f465b7fe640 10 client.4720.objecter ms_dispatch 0x7f4674000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:38:13.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.198+0000 7f465b7fe640 3 client.4720.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:38:13.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.198+0000 7f465b7fe640 3 client.4720.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:38:13.201 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.198+0000 7f465b7fe640 20 client.4720.objecter dump_active .. 0 homeless 2024-03-25T14:38:13.203 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.200+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f466804ee40 con 0x7f467413f710 2024-03-25T14:38:13.203 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.200+0000 7f465b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:13.203 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.200+0000 7f465b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:38:13.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.431+0000 7f467919d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:13.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.431+0000 7f467919d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:13.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.431+0000 7f467919d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:13.434 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.431+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46740627b0 con 0x7f467413f710 2024-03-25T14:38:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.514+0000 7f465b7fe640 1 -- 172.21.15.26:0/1416496416 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f4668050f40 con 0x7f467413f710 2024-03-25T14:38:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.514+0000 7f465b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:13.518 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.514+0000 7f465b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:38:13.518 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:38:13.518 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:38:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 10 librados: watch_flush enter 2024-03-25T14:38:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 10 librados: watch_flush exit 2024-03-25T14:38:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 20 client.4720.objecter shutdown clearing up homeless session... 2024-03-25T14:38:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 10 client.4720.objecter successfully canceled tick 2024-03-25T14:38:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 >> v1:172.21.15.121:6800/288813066 conn(0x7f466006df10 legacy=0x7f46600703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:13.520 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 10 monclient: shutdown 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 >> v1:172.21.15.121:6789/0 conn(0x7f467413f710 legacy=0x7f46741dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 shutdown_connections 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 >> 172.21.15.26:0/1416496416 conn(0x7f46740b7ea0 msgr2=0x7f4674074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 shutdown_connections 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.516+0000 7f467919d640 1 -- 172.21.15.26:0/1416496416 wait complete. 2024-03-25T14:38:13.521 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:13.517+0000 7f467919d640 1 librados: shutdown 2024-03-25T14:38:13.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:38:13.563 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:32.425 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.566 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:32.429 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.575 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:32.453 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.578 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:32.456 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.587 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:32.449 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.591 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:32.453 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.592 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:32.454 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.599 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:32.477 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:13.666 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:32.528 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:20.339 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:38:20.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.452+0000 7fa08036b640 10 monclient: get_monmap_and_config 2024-03-25T14:38:20.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.452+0000 7fa08036b640 10 monclient: build_initial_monmap 2024-03-25T14:38:20.456 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.452+0000 7fa08036b640 10 monclient: monmap: 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:38:20.454166+0000 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:38:20.454166+0000 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:38:20.457 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 1 Processor -- start 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 1 -- start start 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 10 monclient: init 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 10 monclient(hunting): picked mon.noname-c con 0x7fa0780b4d60 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.453+0000 7fa08036b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0780b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.454+0000 7fa08036b640 10 monclient(hunting): picked mon.noname-a con 0x7fa07814be90 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.454+0000 7fa08036b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780af940 con 0x7fa07814be90 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.454+0000 7fa08036b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b0a70 con 0x7fa0780b4d60 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.454+0000 7fa08036b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b0d50 con 0x7fa0780b1730 2024-03-25T14:38:20.459 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.454+0000 7fa08036b640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:20.460 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.454+0000 7fa08036b640 10 monclient(hunting): authenticate will time out at 2185.557660s 2024-03-25T14:38:20.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.467+0000 7fa07e908640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa07814be90 0x7fa07814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57806/0 (socket says 172.21.15.26:57806) 2024-03-25T14:38:20.471 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.467+0000 7fa07e908640 1 -- 172.21.15.26:0/3817197339 learned_addr learned my addr 172.21.15.26:0/3817197339 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:20.478 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.474+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507576332 0 0) 0x7fa0780af940 con 0x7fa07814be90 2024-03-25T14:38:20.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.474+0000 7fa07d105640 10 monclient(hunting): _negotiate 2024-03-25T14:38:20.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.474+0000 7fa07d105640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:20.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.474+0000 7fa07d105640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:20.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.474+0000 7fa07d105640 10 monclient(hunting): my global_id is 4737 2024-03-25T14:38:20.479 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.474+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa064003520 con 0x7fa07814be90 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1514742038 0 0) 0x7fa064003520 con 0x7fa07814be90 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 >> v1:172.21.15.26:6790/0 conn(0x7fa0780b1730 legacy=0x7fa0780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 >> v1:172.21.15.121:6789/0 conn(0x7fa0780b4d60 legacy=0x7fa07813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: _finish_hunting 0 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 1 monclient: found mon.noname-a 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0780b1030 con 0x7fa07814be90 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: _finish_auth 0 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: _check_auth_tickets 2024-03-25T14:38:20.490 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa070001a10 con 0x7fa07814be90 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa08036b640 5 monclient: authenticate success, global_id 4737 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa08036b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa07d105640 10 monclient: dump: 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:20.491 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:20.492 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:20.492 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.485+0000 7fa08036b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:20.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa070001b70 con 0x7fa07814be90 2024-03-25T14:38:20.500 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa07d105640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa07d105640 1 -- 172.21.15.26:0/3817197339 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa070001f80 con 0x7fa07814be90 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa08036b640 10 monclient: get_monmap_and_config success 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa08036b640 4 set_mon_vals no callback set 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa07d105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa07d105640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.496+0000 7fa07d105640 10 monclient: dump: 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:20.501 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:20.502 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:20.502 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 10 monclient: shutdown 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 1 -- 172.21.15.26:0/3817197339 >> v1:172.21.15.26:6789/0 conn(0x7fa07814be90 legacy=0x7fa07814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 1 -- 172.21.15.26:0/3817197339 shutdown_connections 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 1 -- 172.21.15.26:0/3817197339 >> 172.21.15.26:0/3817197339 conn(0x7fa0780617a0 msgr2=0x7fa078061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 1 -- 172.21.15.26:0/3817197339 shutdown_connections 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.497+0000 7fa08036b640 1 -- 172.21.15.26:0/3817197339 wait complete. 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 10 monclient: build_initial_monmap 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 10 monclient: monmap: 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:20.504 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 1 librados: starting msgr at 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 1 librados: starting objecter 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 1 Processor -- start 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 1 -- start start 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 1 librados: setting wanted keys 2024-03-25T14:38:20.505 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 1 librados: calling monclient init 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.498+0000 7fa08036b640 10 monclient: init 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0780b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient(hunting): picked mon.noname-a con 0x7fa0780b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient(hunting): picked mon.noname-c con 0x7fa07814be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa078137760 con 0x7fa0780b4d60 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0781387d0 con 0x7fa0780b1730 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa078140380 con 0x7fa07814be90 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:20.506 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.499+0000 7fa08036b640 10 monclient(hunting): authenticate will time out at 2185.603192s 2024-03-25T14:38:20.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.511+0000 7fa07e107640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa0780b1730 0x7fa078140810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:39648/0 (socket says 172.21.15.26:39648) 2024-03-25T14:38:20.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.511+0000 7fa07e107640 1 -- 172.21.15.26:0/4140093405 learned_addr learned my addr 172.21.15.26:0/4140093405 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:20.515 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.511+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa0780b1730 2024-03-25T14:38:20.516 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.511+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:20.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.515+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320406691 0 0) 0x7fa0781387d0 con 0x7fa0780b1730 2024-03-25T14:38:20.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.515+0000 7fa06effd640 10 monclient(hunting): _negotiate 2024-03-25T14:38:20.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.515+0000 7fa06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:20.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.515+0000 7fa06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:20.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.515+0000 7fa06effd640 10 monclient(hunting): my global_id is 4726 2024-03-25T14:38:20.519 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.515+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c004eb0 con 0x7fa0780b1730 2024-03-25T14:38:20.524 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.520+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa0780b4d60 2024-03-25T14:38:20.525 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.520+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:20.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.524+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4087426944 0 0) 0x7fa05c004eb0 con 0x7fa0780b1730 2024-03-25T14:38:20.528 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.524+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0781387d0 con 0x7fa0780b1730 2024-03-25T14:38:20.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.526+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa07814be90 2024-03-25T14:38:20.530 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.526+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:20.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.531+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa074001b50 con 0x7fa0780b1730 2024-03-25T14:38:20.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.531+0000 7fa06effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:20.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.531+0000 7fa06effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:38:20.535 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.531+0000 7fa06effd640 10 monclient(hunting): dump: 2024-03-25T14:38:20.535 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:20.535 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:20.536 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.531+0000 7fa06effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:38:20.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169439764 0 0) 0x7fa078137760 con 0x7fa0780b4d60 2024-03-25T14:38:20.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): _negotiate 2024-03-25T14:38:20.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:20.539 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): my global_id is 4740 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c004eb0 con 0x7fa0780b4d60 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158965825 0 0) 0x7fa078140380 con 0x7fa07814be90 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): _negotiate 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 10 monclient(hunting): my global_id is 4685 2024-03-25T14:38:20.540 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.535+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078137760 con 0x7fa07814be90 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1533200819 0 0) 0x7fa0781387d0 con 0x7fa0780b1730 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 >> v1:172.21.15.26:6790/0 conn(0x7fa07814be90 legacy=0x7fa0781e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 >> v1:172.21.15.26:6789/0 conn(0x7fa0780b4d60 legacy=0x7fa078144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 10 monclient: _finish_hunting 0 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 1 monclient: found mon.b 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:20.545 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078136910 con 0x7fa0780b1730 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 10 monclient: _finish_auth 0 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 10 monclient: _check_auth_tickets 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa06effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 5 monclient: authenticate success, global_id 4726 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 10 monclient: _renew_subs 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa078136b60 con 0x7fa0780b1730 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 10 monclient: _renew_subs 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0781372b0 con 0x7fa0780b1730 2024-03-25T14:38:20.546 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.540+0000 7fa08036b640 1 librados: init done 2024-03-25T14:38:20.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.541+0000 7fa08036b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:20.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.541+0000 7fa08036b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:20.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.541+0000 7fa08036b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:20.547 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.541+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa078140510 con 0x7fa0780b1730 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa074001cb0 con 0x7fa0780b1730 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa06effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fa074003a60 con 0x7fa0780b1730 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa06effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa06effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa06effd640 10 monclient: dump: 2024-03-25T14:38:20.563 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:20.564 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.559+0000 7fa07c904640 4 set_mon_vals no callback set 2024-03-25T14:38:20.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.560+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fa074003c40 con 0x7fa0780b1730 2024-03-25T14:38:20.565 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.560+0000 7fa07d906640 10 monclient: get_auth_request con 0x7fa05c06ddb0 auth_method 0 2024-03-25T14:38:20.566 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.560+0000 7fa06effd640 10 client.4726.objecter ms_handle_connect 0x7fa05c06ddb0 2024-03-25T14:38:20.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.562+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7fa0740549e0 con 0x7fa0780b1730 2024-03-25T14:38:20.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.562+0000 7fa06effd640 10 client.4726.objecter ms_dispatch 0x7fa078000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:38:20.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.562+0000 7fa06effd640 3 client.4726.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:38:20.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.562+0000 7fa06effd640 3 client.4726.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:38:20.567 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.562+0000 7fa06effd640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T14:38:20.823 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.819+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa074050320 con 0x7fa0780b1730 2024-03-25T14:38:20.823 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.819+0000 7fa06effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:20.823 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:20.819+0000 7fa06effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:38:21.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.072+0000 7fa08036b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:21.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.072+0000 7fa08036b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:21.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.072+0000 7fa08036b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:21.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.072+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0781375a0 con 0x7fa0780b1730 2024-03-25T14:38:21.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.092+0000 7fa06effd640 1 -- 172.21.15.26:0/4140093405 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7fa074052420 con 0x7fa0780b1730 2024-03-25T14:38:21.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.092+0000 7fa06effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:21.096 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.092+0000 7fa06effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:38:21.096 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:38:21.097 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:38:21.098 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.093+0000 7fa08036b640 10 librados: watch_flush enter 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 10 librados: watch_flush exit 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 10 client.4726.objecter successfully canceled tick 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 >> v1:172.21.15.121:6800/288813066 conn(0x7fa05c06ddb0 legacy=0x7fa05c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 10 monclient: shutdown 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 >> v1:172.21.15.121:6789/0 conn(0x7fa0780b1730 legacy=0x7fa078140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 shutdown_connections 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 >> 172.21.15.26:0/4140093405 conn(0x7fa0780617a0 msgr2=0x7fa0780b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 shutdown_connections 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 -- 172.21.15.26:0/4140093405 wait complete. 2024-03-25T14:38:21.099 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:21.094+0000 7fa08036b640 1 librados: shutdown 2024-03-25T14:38:21.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:38:22.910 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:41.772 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:22.915 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:41.777 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:22.945 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:11:41.807 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:22.962 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:41.840 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:22.970 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:41.848 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:22.972 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:41.834 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:22.976 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:41.838 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:23.001 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:11:41.879 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:23.002 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:11:41.864 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:27.916 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:38:28.031 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.027+0000 7fe5be54b640 10 monclient: get_monmap_and_config 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.027+0000 7fe5be54b640 10 monclient: build_initial_monmap 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.027+0000 7fe5be54b640 10 monclient: monmap: 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:38:28.029439+0000 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:38:28.029439+0000 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:38:28.032 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:28.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.028+0000 7fe5be54b640 1 Processor -- start 2024-03-25T14:38:28.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.028+0000 7fe5be54b640 1 -- start start 2024-03-25T14:38:28.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.028+0000 7fe5be54b640 10 monclient: init 2024-03-25T14:38:28.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:28.034 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b80b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b80b1730 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b814be90 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b80af940 con 0x7fe5b80b4d60 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b80b0a70 con 0x7fe5b80b1730 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b80b0d50 con 0x7fe5b814be90 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:28.035 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.029+0000 7fe5be54b640 10 monclient(hunting): authenticate will time out at 2193.133350s 2024-03-25T14:38:28.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.035+0000 7fe5b7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe5b80b4d60 0x7fe5b813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:46422/0 (socket says 172.21.15.26:46422) 2024-03-25T14:38:28.039 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.035+0000 7fe5b7fff640 1 -- 172.21.15.26:0/2248002996 learned_addr learned my addr 172.21.15.26:0/2248002996 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:28.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.039+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412934322 0 0) 0x7fe5b80af940 con 0x7fe5b80b4d60 2024-03-25T14:38:28.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.039+0000 7fe5b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:38:28.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.039+0000 7fe5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:28.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.039+0000 7fe5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:28.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.039+0000 7fe5b6ffd640 10 monclient(hunting): my global_id is 4743 2024-03-25T14:38:28.043 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.039+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a0003520 con 0x7fe5b80b4d60 2024-03-25T14:38:28.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162865896 0 0) 0x7fe5a0003520 con 0x7fe5b80b4d60 2024-03-25T14:38:28.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 >> v1:172.21.15.26:6790/0 conn(0x7fe5b814be90 legacy=0x7fe5b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 >> v1:172.21.15.121:6789/0 conn(0x7fe5b80b1730 legacy=0x7fe5b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:38:28.050 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 1 monclient: found mon.noname-a 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b80b1030 con 0x7fe5b80b4d60 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5be54b640 5 monclient: authenticate success, global_id 4743 2024-03-25T14:38:28.051 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.046+0000 7fe5be54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.050+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe5a8001690 con 0x7fe5b80b4d60 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.050+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.050+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.050+0000 7fe5b6ffd640 10 monclient: dump: 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:28.054 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:28.055 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:28.055 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:28.055 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:28.055 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.050+0000 7fe5be54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5a8001920 con 0x7fe5b80b4d60 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5b6ffd640 1 -- 172.21.15.26:0/2248002996 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe5a8001c30 con 0x7fe5b80b4d60 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5b6ffd640 10 monclient: dump: 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:28.071 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5be54b640 10 monclient: get_monmap_and_config success 2024-03-25T14:38:28.072 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.066+0000 7fe5be54b640 4 set_mon_vals no callback set 2024-03-25T14:38:28.074 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 10 monclient: shutdown 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 1 -- 172.21.15.26:0/2248002996 >> v1:172.21.15.26:6789/0 conn(0x7fe5b80b4d60 legacy=0x7fe5b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 1 -- 172.21.15.26:0/2248002996 shutdown_connections 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 1 -- 172.21.15.26:0/2248002996 >> 172.21.15.26:0/2248002996 conn(0x7fe5b80617a0 msgr2=0x7fe5b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 1 -- 172.21.15.26:0/2248002996 shutdown_connections 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.067+0000 7fe5be54b640 1 -- 172.21.15.26:0/2248002996 wait complete. 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.068+0000 7fe5be54b640 10 monclient: build_initial_monmap 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.068+0000 7fe5be54b640 10 monclient: monmap: 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:38:28.075 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 1 librados: starting msgr at 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 1 librados: starting objecter 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 1 Processor -- start 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 1 -- start start 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 1 librados: setting wanted keys 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 1 librados: calling monclient init 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 10 monclient: init 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:28.076 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.069+0000 7fe5be54b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b80b1730 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b80b4d60 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b814be90 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b81386f0 con 0x7fe5b80b4d60 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b81378c0 con 0x7fe5b814be90 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b8138940 con 0x7fe5b80b1730 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:28.077 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.070+0000 7fe5be54b640 10 monclient(hunting): authenticate will time out at 2193.173692s 2024-03-25T14:38:28.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.078+0000 7fe5bcae8640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe5b814be90 0x7fe5b81d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:60312/0 (socket says 172.21.15.26:60312) 2024-03-25T14:38:28.082 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.078+0000 7fe5bcae8640 1 -- 172.21.15.26:0/1094154693 learned_addr learned my addr 172.21.15.26:0/1094154693 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:28.084 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.080+0000 7fe5b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5b814be90 2024-03-25T14:38:28.084 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.080+0000 7fe5b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:28.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.083+0000 7fe5b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5b80b4d60 2024-03-25T14:38:28.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.083+0000 7fe5b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:28.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.083+0000 7fe5b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5b80b1730 2024-03-25T14:38:28.087 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.083+0000 7fe5b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:28.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.086+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132780703 0 0) 0x7fe5b81378c0 con 0x7fe5b814be90 2024-03-25T14:38:28.090 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.086+0000 7fe5b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:38:28.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.086+0000 7fe5b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:28.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.086+0000 7fe5b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:28.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.086+0000 7fe5b4ff9640 10 monclient(hunting): my global_id is 4732 2024-03-25T14:38:28.091 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.086+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a4004eb0 con 0x7fe5b814be90 2024-03-25T14:38:28.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151701253 0 0) 0x7fe5b8138940 con 0x7fe5b80b1730 2024-03-25T14:38:28.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:38:28.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:28.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:28.107 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 10 monclient(hunting): my global_id is 4691 2024-03-25T14:38:28.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5b81378c0 con 0x7fe5b80b1730 2024-03-25T14:38:28.108 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996122704 0 0) 0x7fe5a4004eb0 con 0x7fe5b814be90 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5b8138940 con 0x7fe5b814be90 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe5b0001c30 con 0x7fe5b814be90 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.103+0000 7fe5b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.104+0000 7fe5b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.104+0000 7fe5b4ff9640 10 monclient(hunting): dump: 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:28.109 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:28.110 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:28.110 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:28.110 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:28.110 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:28.110 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.104+0000 7fe5b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:38:28.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.110+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3681516590 0 0) 0x7fe5b8138940 con 0x7fe5b814be90 2024-03-25T14:38:28.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 >> v1:172.21.15.26:6790/0 conn(0x7fe5b80b1730 legacy=0x7fe5b8136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 >> v1:172.21.15.26:6789/0 conn(0x7fe5b80b4d60 legacy=0x7fe5b8142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:38:28.115 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 1 monclient: found mon.b 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b8143270 con 0x7fe5b814be90 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 10 monclient: _finish_auth 0 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 5 monclient: authenticate success, global_id 4732 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 10 monclient: _renew_subs 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b8137270 con 0x7fe5b814be90 2024-03-25T14:38:28.116 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:38:28.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 10 monclient: _renew_subs 2024-03-25T14:38:28.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:28.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b81d8360 con 0x7fe5b814be90 2024-03-25T14:38:28.117 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.111+0000 7fe5be54b640 1 librados: init done 2024-03-25T14:38:28.118 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.112+0000 7fe5be54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:28.118 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.112+0000 7fe5be54b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:28.118 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.112+0000 7fe5be54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:28.118 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.112+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b80627b0 con 0x7fe5b814be90 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5b0003970 con 0x7fe5b814be90 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7fe5b0003d00 con 0x7fe5b814be90 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b4ff9640 10 monclient: dump: 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:28.385 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:28.386 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.380+0000 7fe5b67fc640 4 set_mon_vals no callback set 2024-03-25T14:38:28.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.381+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7fe5b0003f20 con 0x7fe5b814be90 2024-03-25T14:38:28.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.382+0000 7fe5b7fff640 10 monclient: get_auth_request con 0x7fe5a406dd10 auth_method 0 2024-03-25T14:38:28.387 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.382+0000 7fe5b4ff9640 10 client.4732.objecter ms_handle_connect 0x7fe5a406dd10 2024-03-25T14:38:28.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.387+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7fe5b0054930 con 0x7fe5b814be90 2024-03-25T14:38:28.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.387+0000 7fe5b4ff9640 10 client.4732.objecter ms_dispatch 0x7fe5b8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:38:28.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.387+0000 7fe5b4ff9640 3 client.4732.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:38:28.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.387+0000 7fe5b4ff9640 3 client.4732.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:38:28.391 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.387+0000 7fe5b4ff9640 20 client.4732.objecter dump_active .. 0 homeless 2024-03-25T14:38:28.392 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.388+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5b0050440 con 0x7fe5b814be90 2024-03-25T14:38:28.392 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.388+0000 7fe5b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:28.392 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.388+0000 7fe5b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:38:28.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.649+0000 7fe5be54b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:28.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.649+0000 7fe5be54b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:28.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.649+0000 7fe5be54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:28.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.649+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5b81d78b0 con 0x7fe5b814be90 2024-03-25T14:38:28.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.677+0000 7fe5b4ff9640 1 -- 172.21.15.26:0/1094154693 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7fe5b0052370 con 0x7fe5b814be90 2024-03-25T14:38:28.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.677+0000 7fe5b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:28.682 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.677+0000 7fe5b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:38:28.682 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:38:28.682 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:38:28.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.679+0000 7fe5be54b640 10 librados: watch_flush enter 2024-03-25T14:38:28.684 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 10 librados: watch_flush exit 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 20 client.4732.objecter shutdown clearing up homeless session... 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 10 client.4732.objecter successfully canceled tick 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 >> v1:172.21.15.121:6800/288813066 conn(0x7fe5a406dd10 legacy=0x7fe5a40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 10 monclient: shutdown 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 >> v1:172.21.15.121:6789/0 conn(0x7fe5b814be90 legacy=0x7fe5b81d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 shutdown_connections 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 >> 172.21.15.26:0/1094154693 conn(0x7fe5b80617a0 msgr2=0x7fe5b80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 shutdown_connections 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 -- 172.21.15.26:0/1094154693 wait complete. 2024-03-25T14:38:28.685 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:28.680+0000 7fe5be54b640 1 librados: shutdown 2024-03-25T14:38:28.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:38:35.503 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:38:35.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.619+0000 7f4e277a3640 10 monclient: get_monmap_and_config 2024-03-25T14:38:35.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.619+0000 7f4e277a3640 10 monclient: build_initial_monmap 2024-03-25T14:38:35.622 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.619+0000 7f4e277a3640 10 monclient: monmap: 2024-03-25T14:38:35.622 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:35.622 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:35.622 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:38:35.620038+0000 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:38:35.620038+0000 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-03-25T14:38:35.623 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 1 Processor -- start 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 1 -- start start 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 10 monclient: init 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4e200a98f0 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:35.624 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2013f710 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:35.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4e201430c0 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:35.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.620+0000 7f4e277a3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e20142520 con 0x7f4e200a98f0 2024-03-25T14:38:35.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.621+0000 7f4e277a3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200b28a0 con 0x7f4e201430c0 2024-03-25T14:38:35.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.621+0000 7f4e277a3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200b2a30 con 0x7f4e2013f710 2024-03-25T14:38:35.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.621+0000 7f4e277a3640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:35.625 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.621+0000 7f4e277a3640 10 monclient(hunting): authenticate will time out at 2200.723624s 2024-03-25T14:38:35.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.631+0000 7f4e2553f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4e200a98f0 0x7f4e200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51066/0 (socket says 172.21.15.26:51066) 2024-03-25T14:38:35.634 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.631+0000 7f4e2553f640 1 -- 172.21.15.26:0/2945535851 learned_addr learned my addr 172.21.15.26:0/2945535851 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:35.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.640+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873018288 0 0) 0x7f4e200b28a0 con 0x7f4e201430c0 2024-03-25T14:38:35.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.640+0000 7f4e17fff640 10 monclient(hunting): _negotiate 2024-03-25T14:38:35.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.640+0000 7f4e17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:35.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.640+0000 7f4e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:35.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.640+0000 7f4e17fff640 10 monclient(hunting): my global_id is 4735 2024-03-25T14:38:35.643 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.640+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c003520 con 0x7f4e201430c0 2024-03-25T14:38:35.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.644+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019430885 0 0) 0x7f4e20142520 con 0x7f4e200a98f0 2024-03-25T14:38:35.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.644+0000 7f4e17fff640 10 monclient(hunting): _negotiate 2024-03-25T14:38:35.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.644+0000 7f4e17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:35.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.644+0000 7f4e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:35.647 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.644+0000 7f4e17fff640 10 monclient(hunting): my global_id is 4749 2024-03-25T14:38:35.648 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.644+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e200b28a0 con 0x7f4e200a98f0 2024-03-25T14:38:35.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136601380 0 0) 0x7f4e0c003520 con 0x7f4e201430c0 2024-03-25T14:38:35.651 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 >> v1:172.21.15.26:6790/0 conn(0x7f4e2013f710 legacy=0x7f4e201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 >> v1:172.21.15.26:6789/0 conn(0x7f4e200a98f0 legacy=0x7f4e200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 10 monclient: _finish_hunting 0 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 1 monclient: found mon.noname-c 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200b44c0 con 0x7f4e201430c0 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 10 monclient: _finish_auth 0 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 10 monclient: _check_auth_tickets 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e277a3640 5 monclient: authenticate success, global_id 4735 2024-03-25T14:38:35.652 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.648+0000 7f4e277a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.649+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4e180019d0 con 0x7f4e201430c0 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.649+0000 7f4e17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.649+0000 7f4e17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.649+0000 7f4e17fff640 10 monclient: dump: 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:35.653 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:35.654 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.649+0000 7f4e277a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:38:35.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e18001c20 con 0x7f4e201430c0 2024-03-25T14:38:35.663 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e17fff640 1 -- 172.21.15.26:0/2945535851 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4e18001fb0 con 0x7f4e201430c0 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e277a3640 10 monclient: get_monmap_and_config success 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e277a3640 4 set_mon_vals no callback set 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.660+0000 7f4e17fff640 10 monclient: dump: 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:35.664 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:35.665 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:35.665 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.661+0000 7f4e277a3640 10 monclient: shutdown 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.661+0000 7f4e277a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.661+0000 7f4e277a3640 1 -- 172.21.15.26:0/2945535851 >> v1:172.21.15.121:6789/0 conn(0x7f4e201430c0 legacy=0x7f4e2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.661+0000 7f4e277a3640 1 -- 172.21.15.26:0/2945535851 shutdown_connections 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.661+0000 7f4e277a3640 1 -- 172.21.15.26:0/2945535851 >> 172.21.15.26:0/2945535851 conn(0x7f4e200b7ea0 msgr2=0x7f4e200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.662+0000 7f4e277a3640 1 -- 172.21.15.26:0/2945535851 shutdown_connections 2024-03-25T14:38:35.666 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.662+0000 7f4e277a3640 1 -- 172.21.15.26:0/2945535851 wait complete. 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.663+0000 7f4e277a3640 10 monclient: build_initial_monmap 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.663+0000 7f4e277a3640 10 monclient: monmap: 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-03-25T14:38:35.667 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-c 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.663+0000 7f4e277a3640 1 librados: starting msgr at 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.663+0000 7f4e277a3640 1 librados: starting objecter 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.663+0000 7f4e277a3640 1 Processor -- start 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 1 -- start start 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 1 librados: setting wanted keys 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 1 librados: calling monclient init 2024-03-25T14:38:35.668 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 10 monclient: init 2024-03-25T14:38:35.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 10 monclient: _reopen_session rank -1 2024-03-25T14:38:35.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:38:35.669 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4e200a98f0 addr v1:172.21.15.26:6790/0 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2013f710 addr v1:172.21.15.121:6789/0 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.664+0000 7f4e277a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4e201430c0 addr v1:172.21.15.26:6789/0 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.665+0000 7f4e277a3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e201db9c0 con 0x7f4e201430c0 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.665+0000 7f4e277a3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2006f070 con 0x7f4e2013f710 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.665+0000 7f4e277a3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200701e0 con 0x7f4e200a98f0 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.665+0000 7f4e277a3640 10 monclient(hunting): _renew_subs 2024-03-25T14:38:35.670 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.665+0000 7f4e277a3640 10 monclient(hunting): authenticate will time out at 2200.767691s 2024-03-25T14:38:35.674 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.671+0000 7f4e24d3e640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4e2013f710 0x7f4e201dbe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.26:36522/0 (socket says 172.21.15.26:36522) 2024-03-25T14:38:35.674 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.671+0000 7f4e24d3e640 1 -- 172.21.15.26:0/2573210375 learned_addr learned my addr 172.21.15.26:0/2573210375 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-25T14:38:35.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.673+0000 7f4e15ffb640 10 client.?.objecter ms_handle_connect 0x7f4e2013f710 2024-03-25T14:38:35.676 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.673+0000 7f4e15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:35.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.678+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768110822 0 0) 0x7f4e2006f070 con 0x7f4e2013f710 2024-03-25T14:38:35.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.678+0000 7f4e15ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:38:35.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.678+0000 7f4e15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:35.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.678+0000 7f4e15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:35.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.678+0000 7f4e15ffb640 10 monclient(hunting): my global_id is 4738 2024-03-25T14:38:35.681 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.678+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e04004eb0 con 0x7f4e2013f710 2024-03-25T14:38:35.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.684+0000 7f4e15ffb640 10 client.?.objecter ms_handle_connect 0x7f4e200a98f0 2024-03-25T14:38:35.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.684+0000 7f4e15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:35.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.684+0000 7f4e15ffb640 10 client.?.objecter ms_handle_connect 0x7f4e201430c0 2024-03-25T14:38:35.687 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.684+0000 7f4e15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1095030821 0 0) 0x7f4e04004eb0 con 0x7f4e2013f710 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e2006f070 con 0x7f4e2013f710 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4e100019d0 con 0x7f4e2013f710 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 10 monclient(hunting): dump: 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:35.692 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:35.693 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:35.693 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:35.693 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:35.693 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:35.693 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.688+0000 7f4e15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:38:35.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.690+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533651164 0 0) 0x7f4e201db9c0 con 0x7f4e201430c0 2024-03-25T14:38:35.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.690+0000 7f4e15ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:38:35.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.690+0000 7f4e15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:38:35.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.690+0000 7f4e15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:38:35.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.690+0000 7f4e15ffb640 10 monclient(hunting): my global_id is 4752 2024-03-25T14:38:35.694 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.690+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e04004eb0 con 0x7f4e201430c0 2024-03-25T14:38:35.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2885483431 0 0) 0x7f4e2006f070 con 0x7f4e2013f710 2024-03-25T14:38:35.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 >> v1:172.21.15.26:6790/0 conn(0x7f4e200a98f0 legacy=0x7f4e2006e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:35.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 >> v1:172.21.15.26:6789/0 conn(0x7f4e201430c0 legacy=0x7f4e201e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:35.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:38:35.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 1 monclient: found mon.b 2024-03-25T14:38:35.701 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2006dea0 con 0x7f4e2013f710 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 10 monclient: _finish_auth 0 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 5 monclient: authenticate success, global_id 4738 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 10 monclient: _renew_subs 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2006f2a0 con 0x7f4e2013f710 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 10 monclient: _renew_subs 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:35.702 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e201dc560 con 0x7f4e2013f710 2024-03-25T14:38:35.703 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.698+0000 7f4e277a3640 1 librados: init done 2024-03-25T14:38:35.703 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.699+0000 7f4e277a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:35.703 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.699+0000 7f4e277a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:35.703 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.699+0000 7f4e277a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:35.703 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.699+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e201dbb70 con 0x7f4e2013f710 2024-03-25T14:38:35.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e10001c50 con 0x7f4e2013f710 2024-03-25T14:38:35.967 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 486473287 0 0) 0x7f4e100038c0 con 0x7f4e2013f710 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e15ffb640 10 monclient: dump: 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:fsid 3ca971d3-87b7-4101-bfd9-2dfef6094512 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:created 2024-03-25T14:26:10.106225+0000 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 3 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-25T14:38:35.968 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.c 2024-03-25T14:38:35.969 INFO:teuthology.misc.health.smithi026.stderr: 2024-03-25T14:38:35.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.964+0000 7f4e177fe640 4 set_mon_vals no callback set 2024-03-25T14:38:35.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.965+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 720463947 0 0) 0x7f4e10003ae0 con 0x7f4e2013f710 2024-03-25T14:38:35.969 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.966+0000 7f4e2553f640 10 monclient: get_auth_request con 0x7f4e0406ddd0 auth_method 0 2024-03-25T14:38:35.970 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.966+0000 7f4e15ffb640 10 client.4738.objecter ms_handle_connect 0x7f4e0406ddd0 2024-03-25T14:38:35.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3094416816 0 0) 0x7f4e10054670 con 0x7f4e2013f710 2024-03-25T14:38:35.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 10 client.4738.objecter ms_dispatch 0x7f4e20000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:38:35.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:38:35.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:38:35.972 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T14:38:35.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e100b18b0 con 0x7f4e2013f710 2024-03-25T14:38:35.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:38:35.973 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:35.969+0000 7f4e15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:38:36.204 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.200+0000 7f4e277a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:36.204 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.200+0000 7f4e277a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:36.204 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.200+0000 7f4e277a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-25T14:38:36.204 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.200+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e200627b0 con 0x7f4e2013f710 2024-03-25T14:38:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.223+0000 7f4e15ffb640 1 -- 172.21.15.26:0/2573210375 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 684585473) 0x7f4e10050180 con 0x7f4e2013f710 2024-03-25T14:38:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.223+0000 7f4e15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:38:36.226 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.223+0000 7f4e15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:38:36.226 INFO:teuthology.misc.health.smithi026.stdout: 2024-03-25T14:38:36.227 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"3ca971d3-87b7-4101-bfd9-2dfef6094512","created":"2024-03-25T14:26:58.489001+0000","modified":"2024-03-25T14:37:02.542800+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:37:02.542800+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d5de21c8-0a56-4aa3-a772-897f8c8c05db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5620bbc5-68e9-4f64-b0ae-62568316e93e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5536078d-b3de-4699-8956-d3f8248879ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"558cf5ab-35b6-42a3-9e68-1d17ee5dae34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3c91e75-ead7-4d38-9c87-b7a3105fdff3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"51fdbe3a-8a98-458e-b558-7304df69e600","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f32d3668-ed36-49b2-a983-3d512dc5f96f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ac269fe1-ef8f-4fe9-80e6-3081e6131db8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:38:36.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.225+0000 7f4e277a3640 10 librados: watch_flush enter 2024-03-25T14:38:36.229 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.225+0000 7f4e277a3640 10 librados: watch_flush exit 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.225+0000 7f4e277a3640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.225+0000 7f4e277a3640 10 client.4738.objecter successfully canceled tick 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 >> v1:172.21.15.121:6800/288813066 conn(0x7f4e0406ddd0 legacy=0x7f4e04070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 10 monclient: shutdown 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 >> v1:172.21.15.121:6789/0 conn(0x7f4e2013f710 legacy=0x7f4e201dbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 shutdown_connections 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 >> 172.21.15.26:0/2573210375 conn(0x7f4e200b7ea0 msgr2=0x7f4e2014e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 shutdown_connections 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 -- 172.21.15.26:0/2573210375 wait complete. 2024-03-25T14:38:36.230 INFO:teuthology.misc.health.smithi026.stderr:2024-03-25T14:38:36.226+0000 7f4e277a3640 1 librados: shutdown 2024-03-25T14:38:36.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:38:36.240 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T14:38:36.242 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T14:38:36.243 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T14:38:36.243 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:38:38.162 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:11:33.327 35702== 2024-03-25T14:38:38.163 INFO:tasks.ceph.osd.0.smithi026.stderr:==00:00:11:33.327 35702== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:38:38.197 INFO:tasks.ceph.osd.0.smithi026.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:38:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:38:42.344 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T14:38:42.345 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T14:38:42.346 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T14:38:42.346 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:38:42.958 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:01.821 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:42.965 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:01.827 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.008 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:01.870 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.009 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:01.887 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.013 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:01.891 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.021 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:01.884 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.033 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:01.911 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.035 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:01.898 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:43.063 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:01.925 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:38:44.376 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:11:39.522 35705== 2024-03-25T14:38:44.376 INFO:tasks.ceph.osd.1.smithi026.stderr:==00:00:11:39.522 35705== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:38:44.417 INFO:tasks.ceph.osd.1.smithi026.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:38:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:38:48.448 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T14:38:48.448 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T14:38:48.449 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T14:38:48.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:38:50.414 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:11:45.559 35706== 2024-03-25T14:38:50.414 INFO:tasks.ceph.osd.2.smithi026.stderr:==00:00:11:45.559 35706== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:38:50.450 INFO:tasks.ceph.osd.2.smithi026.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:38:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:38:54.551 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T14:38:54.551 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T14:38:54.552 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T14:38:54.552 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:38:56.335 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:11:51.482 35704== 2024-03-25T14:38:56.335 INFO:tasks.ceph.osd.3.smithi026.stderr:==00:00:11:51.482 35704== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:38:56.375 INFO:tasks.ceph.osd.3.smithi026.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:00.654 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T14:39:00.654 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T14:39:00.654 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T14:39:00.655 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:02.580 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:11:57.714 35318== 2024-03-25T14:39:02.581 INFO:tasks.ceph.osd.4.smithi121.stderr:==00:00:11:57.714 35318== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:02.617 INFO:tasks.ceph.osd.4.smithi121.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:03.076 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:21.938 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.119 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:21.997 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.122 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:22.000 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.127 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:21.990 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.145 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:22.008 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.150 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:22.029 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.170 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:22.033 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.174 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:22.037 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:03.202 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:22.065 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:06.757 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T14:39:06.757 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T14:39:06.758 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T14:39:06.758 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:08.730 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:12:03.863 35319== 2024-03-25T14:39:08.730 INFO:tasks.ceph.osd.5.smithi121.stderr:==00:00:12:03.863 35319== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:08.771 INFO:tasks.ceph.osd.5.smithi121.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:12.859 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T14:39:12.860 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T14:39:12.860 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T14:39:12.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:14.519 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:12:09.650 35320== 2024-03-25T14:39:14.519 INFO:tasks.ceph.osd.6.smithi121.stderr:==00:00:12:09.650 35320== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:14.559 INFO:tasks.ceph.osd.6.smithi121.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:18.961 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:18.962 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T14:39:18.962 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T14:39:18.962 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T14:39:18.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:20.404 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:12:15.532 35321== 2024-03-25T14:39:20.404 INFO:tasks.ceph.osd.7.smithi121.stderr:==00:00:12:15.532 35321== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:20.441 INFO:tasks.ceph.osd.7.smithi121.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:23.025 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:41.887 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.027 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:41.890 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.043 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:41.905 35388== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.072 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:41.951 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.074 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:41.952 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.084 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:41.963 34741== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.101 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:41.964 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.103 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:41.966 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:23.118 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:41.981 35391== Warning: unimplemented fcntl command: 1036 2024-03-25T14:39:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:25.064 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T14:39:25.065 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T14:39:25.065 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T14:39:25.065 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T14:39:25.065 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:25.089 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T14:39:25.090 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T14:39:25.090 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:25.119 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T14:39:25.119 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T14:39:25.119 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T14:39:25.119 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:25.145 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-03-25T14:39:25.131+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35357) UID: 0 2024-03-25T14:39:25.145 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-03-25T14:39:25.133+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T14:39:25.631 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:44.493 35388== 2024-03-25T14:39:25.631 INFO:tasks.ceph.mon.a.smithi026.stderr:==00:00:12:44.493 35388== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:25.640 INFO:tasks.ceph.mon.a.smithi026.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:31.221 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T14:39:31.221 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T14:39:31.222 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T14:39:31.222 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:31.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-03-25T14:39:31.222+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35359) UID: 0 2024-03-25T14:39:31.238 INFO:tasks.ceph.mon.c.smithi026.stderr:2024-03-25T14:39:31.224+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T14:39:31.884 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:50.746 35391== 2024-03-25T14:39:31.884 INFO:tasks.ceph.mon.c.smithi026.stderr:==00:00:12:50.746 35391== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:31.893 INFO:tasks.ceph.mon.c.smithi026.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:37.324 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T14:39:37.324 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T14:39:37.325 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T14:39:37.325 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:39:37.345 INFO:tasks.ceph.mon.b.smithi121.stderr:2024-03-25T14:39:37.326+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34737) UID: 0 2024-03-25T14:39:37.346 INFO:tasks.ceph.mon.b.smithi121.stderr:2024-03-25T14:39:37.328+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T14:39:37.764 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:56.643 34741== 2024-03-25T14:39:37.764 INFO:tasks.ceph.mon.b.smithi121.stderr:==00:00:12:56.643 34741== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:39:37.773 INFO:tasks.ceph.mon.b.smithi121.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:39:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:39:43.426 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T14:39:43.426 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T14:39:43.427 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T14:39:43.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T14:39:43.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:39:43.474 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T14:39:43.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:39:43.602 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T14:39:43.686 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:39:43.686 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T14:39:43.767 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:39:43.767 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T14:39:43.834 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:39:43.834 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T14:39:43.894 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:39:43.894 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T14:39:44.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:39:44.016 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T14:39:44.075 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:39:44.075 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T14:39:44.132 INFO:tasks.ceph:Archiving mon data... 2024-03-25T14:39:44.134 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/data/mon.a.tgz 2024-03-25T14:39:44.134 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-25T14:39:44.151 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.jlfepFar2G 2024-03-25T14:39:44.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.jlfepFar2G 2024-03-25T14:39:44.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.jlfepFar2G 2024-03-25T14:39:44.343 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.jlfepFar2G is 133KB 2024-03-25T14:39:44.396 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.jlfepFar2G 2024-03-25T14:39:44.411 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/data/mon.c.tgz 2024-03-25T14:39:44.411 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-25T14:39:44.464 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.dxuCdtlIgr 2024-03-25T14:39:44.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dxuCdtlIgr 2024-03-25T14:39:44.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.dxuCdtlIgr 2024-03-25T14:39:44.640 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.dxuCdtlIgr is 133KB 2024-03-25T14:39:44.691 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.dxuCdtlIgr 2024-03-25T14:39:44.706 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/data/mon.b.tgz 2024-03-25T14:39:44.706 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-03-25T14:39:44.724 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.6iCZRbACcA 2024-03-25T14:39:44.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6iCZRbACcA 2024-03-25T14:39:44.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.6iCZRbACcA 2024-03-25T14:39:44.919 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.6iCZRbACcA is 133KB 2024-03-25T14:39:44.970 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.6iCZRbACcA 2024-03-25T14:39:44.985 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T14:39:44.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T14:39:44.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T14:39:45.379 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T14:39:45.651 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T14:39:45.653 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/remote/smithi026/crash 2024-03-25T14:39:45.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T14:39:45.685 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/remote/smithi121/crash 2024-03-25T14:39:45.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T14:39:45.712 INFO:tasks.ceph:Compressing logs... 2024-03-25T14:39:45.713 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T14:39:45.732 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T14:39:45.759 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-25T14:39:45.760 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40140.log 2024-03-25T14:39:45.760 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35709.log 2024-03-25T14:39:45.760 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40140.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-25T14:39:45.761 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T14:39:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42519.log 2024-03-25T14:39:45.770 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40140.log.gz 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.a.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35709.log.gz 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40803.log 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42519.log.gz 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40959.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40803.log.gz 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40374.log 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40257.log 86.7% 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2024-03-25T14:39:45.771 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40257.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40374.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40725.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40257.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40998.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 86.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/valgrind/osd.2.log 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T14:39:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40023.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40569.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42441.log 2024-03-25T14:39:45.773 INFO:teuthology.orchestra.run.smithi026.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41895.log 86.9% 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40023.log.gz 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42441.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40569.log.gzgzip 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40335.log 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42441.log.gz 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41700.log: gzip -5 86.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.41895.log.gz -- 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.37852.log 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40335.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41037.log 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41778.log 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40335.log.gz 2024-03-25T14:39:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41037.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37852.log.gz 2024-03-25T14:39:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2024-03-25T14:39:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41778.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41037.log.gz 2024-03-25T14:39:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-25T14:39:45.775 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39943.log 2024-03-25T14:39:45.775 INFO:teuthology.orchestra.run.smithi026.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41778.log.gz 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34894.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34894.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34894.log.gz 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/valgrind/osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T14:39:45.798 INFO:teuthology.orchestra.run.smithi121.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34974.log 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35185.log 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34974.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35185.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34974.log.gz 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T14:39:45.799 INFO:teuthology.orchestra.run.smithi121.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35185.log.gz 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.4.log: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35096.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34934.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35096.log: /var/log/ceph/ceph-client.admin.34934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35145.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35096.log.gz 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34934.log.gz 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35145.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35145.log.gz 2024-03-25T14:39:45.800 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T14:39:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39670.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2024-03-25T14:39:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42324.log 2024-03-25T14:39:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39943.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-25T14:39:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39984.log 2024-03-25T14:39:45.814 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41661.log 2024-03-25T14:39:45.814 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39943.log.gz 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39984.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.42324.log.gz 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41661.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.39984.log.gz -5 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36750.log 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41661.log.gz 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41505.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36750.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40062.log 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40101.log 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36750.log.gz 2024-03-25T14:39:45.815 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40062.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40062.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40101.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39278.log 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41544.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39278.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40530.log 86.5% 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39278.log.gz 2024-03-25T14:39:45.816 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40530.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39865.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42246.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40530.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35873.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40491.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39865.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40491.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35873.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40491.log.gz 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.40764.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40686.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2024-03-25T14:39:45.817 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40179.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41154.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40686.log.gz 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- 86.9% /var/log/ceph/ceph-client.admin.42090.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40179.log: /var/log/ceph/ceph-client.admin.41271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42285.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40179.log.gz 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42090.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42090.log.gz 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35732.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42285.log: /var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39826.log 2024-03-25T14:39:45.818 INFO:teuthology.orchestra.run.smithi026.stderr: 85.9%/var/log/ceph/ceph-client.admin.35732.log: -- replaced with /var/log/ceph/ceph-client.admin.42285.log.gz 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39826.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35732.log.gz 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39826.log.gz -5 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39787.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.42129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39787.log: gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz --verbose 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42363.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39787.log.gz 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39397.log: gzip 87.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz --verbose 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.42012.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-25T14:39:45.819 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz/var/log/ceph/ceph-client.admin.42363.log: 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41427.log 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41934.log 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42012.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41427.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41856.log 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.41934.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41427.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35399.log 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41934.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41856.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35399.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-03-25T14:39:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42480.log 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40647.log 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40881.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35399.log.gz 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39631.log 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42480.log.gz 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39631.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40647.log.gz 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42207.log 2024-03-25T14:39:45.821 INFO:teuthology.orchestra.run.smithi026.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41076.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42207.log.gz 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39030.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40608.log 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40608.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T14:39:45.822 INFO:teuthology.orchestra.run.smithi026.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40608.log.gz 2024-03-25T14:39:45.823 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2024-03-25T14:39:45.823 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.0.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-03-25T14:39:45.823 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42051.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41115.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.2.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29536.log 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39514.log: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T14:39:45.824 INFO:teuthology.orchestra.run.smithi026.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.tmp-client.admin.29536.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29536.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41817.log 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41349.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42168.log 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41817.log: /var/log/ceph/ceph-client.admin.42168.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41817.log.gz 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42168.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.41973.log 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.40218.log: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42402.log 2024-03-25T14:39:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.42402.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41973.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40296.log 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42402.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.41388.log: /var/log/ceph/ceph-client.admin.40296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.c.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40296.log.gz 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40842.log 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.39748.log: 86.1%/var/log/ceph/ceph-client.admin.40842.log: -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-03-25T14:39:45.826 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40842.log.gz 2024-03-25T14:39:45.861 INFO:teuthology.orchestra.run.smithi121.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T14:39:46.652 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T14:39:46.683 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T14:39:46.946 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T14:39:46.970 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T14:39:46.973 INFO:teuthology.orchestra.run.smithi026.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T14:39:46.991 INFO:teuthology.orchestra.run.smithi026.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T14:39:46.996 INFO:teuthology.orchestra.run.smithi026.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T14:39:47.018 INFO:teuthology.orchestra.run.smithi026.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T14:39:50.381 INFO:teuthology.orchestra.run.smithi026.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T14:39:50.733 INFO:teuthology.orchestra.run.smithi121.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T14:39:50.735 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-25T14:39:50.735 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m4.968s 2024-03-25T14:39:50.735 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m8.850s 2024-03-25T14:39:50.735 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.322s 2024-03-25T14:39:50.961 INFO:teuthology.orchestra.run.smithi026.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T14:39:50.963 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-25T14:39:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m5.217s 2024-03-25T14:39:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m14.201s 2024-03-25T14:39:50.963 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.544s 2024-03-25T14:39:50.964 INFO:tasks.ceph:Archiving logs... 2024-03-25T14:39:50.964 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/remote/smithi026/log 2024-03-25T14:39:50.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T14:39:53.568 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/remote/smithi121/log 2024-03-25T14:39:53.569 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T14:39:55.173 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T14:39:55.407 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f26eba35863a42b2874936f7951f24b8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T14:39:55.412 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T14:39:55.426 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T14:39:55.437 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T14:39:55.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T14:39:55.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T14:39:55.534 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T14:39:55.542 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T14:39:55.543 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:39:55.545 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:39:55.560 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:39:55.561 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:39:55.564 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:39:55.565 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-25T14:39:55.565 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +1033us[+1033us] +/- 86ms 2024-03-25T14:39:55.565 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1527us[-1547us] +/- 52ms 2024-03-25T14:39:55.565 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +1440us[+1440us] +/- 88ms 2024-03-25T14:39:55.565 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:39:55.568 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:39:55.568 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-25T14:39:55.568 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1033us[+1033us] +/- 86ms 2024-03-25T14:39:55.568 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1525us[-1538us] +/- 52ms 2024-03-25T14:39:55.568 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +1447us[+1447us] +/- 88ms 2024-03-25T14:39:55.569 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:39:55.569 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T14:39:55.579 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T14:39:55.580 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T14:39:55.592 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T14:39:55.680 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T14:39:55.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:39:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:39:55.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:39:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:39:55.815 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T14:39:55.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T14:39:55.838 INFO:teuthology.task.internal:Duration was 1684.558237 seconds 2024-03-25T14:39:55.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T14:39:55.850 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T14:39:55.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T14:39:55.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T14:39:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:39:55.898 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:39:56.277 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T14:39:56.277 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-03-25T14:39:56.277 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T14:39:56.299 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-03-25T14:39:56.299 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T14:39:56.341 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T14:39:56.341 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T14:39:56.344 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T14:39:56.409 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T14:39:56.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T14:39:56.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T14:39:56.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T14:39:56.556 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T14:39:56.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T14:39:56.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T14:39:56.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T14:39:56.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:39:56.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:39:56.647 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-03-25T14:39:56.676 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-03-25T14:39:56.707 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:39:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:39:56.743 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:39:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:39:56.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T14:39:56.772 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T14:39:56.773 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/remote/smithi026 2024-03-25T14:39:56.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T14:39:56.824 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287/remote/smithi121 2024-03-25T14:39:56.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T14:39:56.861 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T14:39:56.861 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T14:39:56.864 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T14:39:56.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T14:39:56.921 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T14:39:56.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T14:39:57.002 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T14:39:57.003 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T14:39:57.005 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T14:39:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 14:39 /home/ubuntu/cephtest 2024-03-25T14:39:57.020 INFO:teuthology.orchestra.run.smithi121.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 14:39 /home/ubuntu/cephtest 2024-03-25T14:39:57.021 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T14:39:57.035 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T14:39:57.057 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T14:39:57.083 DEBUG:teuthology.nuke:shortname: smithi026 2024-03-25T14:39:57.084 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:39:57.108 DEBUG:teuthology.nuke:shortname: smithi121 2024-03-25T14:39:57.109 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:39:57.115 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:58:41.364631', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLLiXIgngNtdUUxjOEdjf74oeU6tS2Dqa8Z23Rg++8KRtAagxxs2CR8ZaYaXIpIn8ME7NRK96dK8IlUl2h3juVg='} 2024-03-25T14:39:57.128 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621287', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:58:41.365932', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvUXbyBTSDKsGDyI5JnMlAwPUD+cGxsn5VWUYR714IAgvBMNY3l3MWwkqHEz6JahOYUCIE5jkKSzVxpToE2Kvc='} 2024-03-25T14:39:57.162 INFO:teuthology.orchestra.console.smithi026:Power off 2024-03-25T14:39:57.162 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:39:57.180 INFO:teuthology.orchestra.console.smithi121:Power off 2024-03-25T14:39:57.180 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:39:57.190 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-03-25T14:39:57.190 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:39:57.203 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-03-25T14:39:57.204 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:39:57.216 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-25T14:39:57.229 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-25T14:40:01.217 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:40:01.229 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:40:01.328 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-25T14:40:01.340 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-25T14:40:05.330 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:40:05.341 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:40:05.441 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-25T14:40:05.450 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-25T14:40:05.542 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-03-25T14:40:09.454 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:40:09.565 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-03-25T14:40:09.666 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-03-25T14:40:09.770 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 1684.5582373142242 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f26eba35863a42b2874936f7951f24b8 status: fail success: false 2024-03-25T14:40:09.771 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T14:40:09.844 INFO:teuthology.run:FAIL